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ABSTRACT 


Marine  Corps  Total  Life  Cycle  Management  (TLCM)  is  critical  in  meeting 
requirements  established  in  Department  of  Defense  Directive  4151.18,  notably, 
“optimizing  ...  concepts  to  deliver  efficient  and  effective  perfonnance  to  the  operating 
forces.”  Modeling  and  simulation  (M&S)  creates  an  opportunity  to  explore  improvement 
opportunities  before  costly  decisions  are  implemented.  Unfortunately,  applying  M&S  to 
TLCM  efforts  has  been  hampered  in  the  past  by  an  inefficient,  error  prone  process  of 
moving  gathered  data  to  an  M&S  platform. 

This  research  uses  Visual  Basic  for  Applications  to  link  two  Marine  Corps  TLCM 
tools:  the  Systems  Operational  Effectiveness  Decision  Support  Tool  (SOE  DST)  and  the 
Total  Life  Cycle  Management  Assessment  Tool  (TLCM-AT).  The  Bridging  Operational 
Logistics  Tool  (B-OLT)  is  created  to  allow  TLCM-AT  models  to  be  built  automatically, 
using  existing  SOE  DST  data  and  limited  subject  matter  expert  inputs. 

The  B-OLT  built  models  are  assessed,  exercised  with  state-of-the-art  design  of 
experiments  and  used  to  predict  future  events. 

The  research  shows  a  link  between  data  currently  collected  and  simulation  allows 
for  quantitative  analysis.  This  analysis  explores  the  Marine  Corps’  data  collection  and 
summary  techniques,  and  their  application  to  modeling,  demonstrating  how  B-OLT  can 
be  used  to  aid  in  future  analytical  efforts. 
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EXECUTIVE  SUMMARY 


Marine  Corps  Total  Life  Cycle  Management  (TLCM)  is  critical  in  meeting  the 
requirements  established  in  Department  of  Defense  Directive  4151.18,  notably, 
“optimizing  .  .  .  operating  concepts  to  deliver  efficient  and  effective  perfonnance  to  the 
operating  forces”  (Wolfowitz,  2004).  Modeling  and  simulation  (M&S)  creates  an 
opportunity  to  explore  improvement  opportunities  before  costly  decisions  are 
implemented.  Applying  M&S  to  TLCM  efforts  has  been  hampered  in  the  past  by  an 
inefficient,  error-prone,  and  laborious  process  of  moving  gathered  data  to  an  M&S 
platform.  This  research  applies  Visual  Basic  for  Applications  code  to  the  problem  of 
migrating  data  gathered  and  summarized  to  a  modeling  environment.  These  models  are 
then  assessed,  used  as  a  predictive  tool,  and  their  sensitivities  to  input  factors  explored. 
Through  automation,  M&S  can  more  readily  be  used  to  explore  program  improvements, 
improve  provisioning  efforts,  and  define  budget  requirements  to  support  maintenance. 

TLCM  is  a  complicated  process.  The  most  powerful  tools  of  the  TLCM 
facilitator  are  good  data  and  simulation.  The  data  collected  provide  information  about  the 
end  item.  Simulation  provides  a  way  to  test  changes  to  the  system  before  costs  are 
incurred.  The  use  of  data  collected  in  simulation,  in  a  process-oriented  way,  makes  M&S 
accessible,  allows  for  easy  implementation  of  design  of  experiments  (DOE),  and  makes 
validation  possible. 

To  be  functional,  the  process  must  be  easily  executed  and  understood.  The  user 
should  also  have  a  reason  to  use  the  process.  This  research  offers  an  approach  to  meeting 
this  process  requirement  by  using  the  following  questions  as  a  guide: 

•  Can  the  development  of  Total  Life  Cycle  Management  Analysis  Tool 
(TLCM-AT)  models  be  aided  through  automation? 

•  What  gaps  are  there  between  data  summarized  in  the  Systems  Operational 
Effectiveness  Decision  Support  Tool  and  data  required  in  TLCM-AT? 

•  Once  a  TLCM-AT  model  is  built,  how  well  does  the  model  assess  against 
reality;  and  what  factors  are  most  relevant? 

•  Given  an  assessed  model  and  known  fluctuations  in  operational  tempo  or 
vehicle  population,  can  the  model  predict  changes  in  parts  failure  events? 
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The  Marine  Corps  has  contracted  two  independent  TLCM  tools:  The  Systems 
Operational  Effectiveness  Decision  Support  Tool  (SOE  DST)  and  TLCM-AT.  These  two 
TLCM  tools  are  not  connected,  causing  their  full  potential  to  go  unrealized.  SOE  DST 
collects  inputs  from  over  12  maintenance  and  supply  user  interfaces.  These  inputs 
pertain  to  critical  end  items  used  throughout  the  Marine  Corps.  SOE  DST  reports 
historical  facts  pertaining  to  these  end  items;  it  is  not  designed  to  act  as  a  predictive  tool. 
TLCM-AT  has  the  ability  to  make  predictions  based  on  projected  operational  tempo, 
distributions  of  failure  rates,  and  logistics  response  times,  but  requires  real-time  data  in 
order  to  produce  accurate  and  relevant  output.  While  SOE  DST  contains  information 
necessary  to  populate  TLCM-AT,  there  is  no  established  interface  between  the  two 
systems. 

It  is  essential  to  have  a  tool  that  can  link  SOE  DST  and  TLCM-AT.  The  link 
should  be  automated  to  ensure  tested  accuracy  of  the  process.  The  result  of  the  link  is  an 
opportunity  to  rapidly  employ  a  predictive  model  with  data  that  is  readily  available  to 
program  managers. 

This  research  challenges  old  constructs  of  TLCM-AT  models  built  by 
modeling  professionals  from  Clockwork  Solutions  using  multiple  sources  of  inputs,  to 
include  SOE  DST.  The  Bridging  Operational  Logistics  Tool  (B-OLT)  was  built  to 
automate  the  model-build  process.  While  licenses  are  available  to  the  Marine  Corps,  the 
model-build  process  is  currently  too  complicated  to  be  functionally  practical.  Prior  to  B- 
OLT,  it  took  a  trained  TLCM-AT  user  approximately  three  days  to  build  a  rudimentary 
model  strictly  from  SOE  DST  with  limited  subject  matter  expert  data.  B-OLT  uses  SOE 
DST  data  to  build,  run,  and  extract  output  from  a  powerful,  closed-loop,  stochastic  model 
in  TLCM-AT  in  less  than  10  minutes.  The  automation  demonstrates  the  ability  to  put 
M&S  in  the  hands  of  program  managers  in  a  user-friendly  way.  Figure  1  demonstrates 
this  process. 
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Figure  1.  The  SOE  to  TLCM-AT  link  developed  by  B-OLT. 


The  automation  itself  saves  time,  makes  modeling  accessible  to  all  Table  of 
Authorize  Materiel  Control  Numbers  reported  by  SOE  DST,  produces  results  from  data 
that  can  be  validated,  and  allows  DOE  to  be  applied  to  gain  insight  for  policy  decision 
making. 

Through  the  process  of  assessment,  ways  to  enhance  SOE  DST  are  discovered. 
The  B-OLT-built  model  reported  within  10%  of  the  same  failures  experienced  in  reality. 
The  proximity  to  reality  makes  the  models  practical  to  use  in  future  ‘what  if  analysis.  In 
the  course  of  this  research,  opportunities  to  improve  SOE  DST  for  use  as  input  into  M&S 
platforms  were  discovered.  There  are  factors  missing  from  SOE  DST  that  are  necessary 
for  TLCM  M&S:  the  average  miles  per  hour  and  miles  driven,  maintenance  and  supply 
times,  indenture  structure  and  the  vehicle  counts  used  to  compute  failure  rates. 
Indentured  structure  will  allow  for  a  more  robust  series  of  models  by  reducing  the  amount 
of  memory  taken  by  the  simulation.  The  remaining  factors  directly  affect  the  outputs 
from  the  model. 

With  B-OLT,  multiple  models  can  be  built  and  executed  in  sequence,  allowing  a 
DOE  application  to  determine  input  factors  of  interest.  The  measures  of  effectiveness 
can  be  chosen  from  any  number  of  outputs,  such  as  cost  or  performance.  This  research 
used  modeled  versus  real  failures  as  the  measure  of  effectiveness.  Through  DOE,  it  was 
discovered  that  the  failure  rate  and  vehicle  counts  are  the  most  important  consideration  in 
terms  of  data  inputs.  Knowing  this  gives  focus  of  effort  when  considering  where  to 
improve  data  collection  in  the  future. 

Once  the  B-OLT  model-building  process  is  assessed,  it  is  used  to  predict  failures. 
An  automated  model  built  from  2002  data  is  used  to  predict  2003  requirements.  The  First 
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Marine  Expeditionary  Unit  is  used  as  the  test  bed.  From  2002  to  2003,  the  unit’s 
operational  tempo  and  vehicle  populations  increased  in  response  to  mission  requirements. 
Using  the  future  increase  in  operational  tempo  and  population,  coupled  with  2002  SOE 
DST  data  only,  models  were  built  and  run.  The  models  were  able  to  accurately  predict 
future  requirements.  If  the  model  had  been  used  to  build  a  provisioning  package,  the  unit 
would  have  enjoyed  favorable  results.  Specifically,  the  results  show  that  going  one 
standard  deviation  more  than  the  model’s  prediction  would  have  resulted  in  being  short 
on  only  17  of  297  parts  during  the  one-year  provisioning  period. 

There  are  important  lessons  learned  that  must  be  addressed  to  improve  the  overall 
TLCM  data  gathering  to  simulation  process;  however,  this  research  provides  a 
demonstrated  capability  to  move  from  data  that  is  already  collected  and  summarized  to  a 
predictive  model  in  an  automated  manner.  By  automating  many  human-in-the-loop 
activities,  variability  and  bias  is  removed  from  the  models  created.  Further,  the  process, 
and  its  associated  data,  can  be  validated  over  time.  This  automation  can  lead  to 
validation  of  the  model-build  process  and  put  modeling  in  the  hands  of  PMs  to  assist  in 
policy  decisions. 
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I.  MARINE  CORPS  MAINTENANCE  IS  COMMITTED  TO 
SUPPORTING  THE  WARFIGHTER 


Marine  Corps  Total  Life  Cycle  Management  (TLCM)  is  critical  in  meeting  the 
requirements  established  in  Department  of  Defense  (DoD)  Directive  4151.18,  notably, 
“optimizing  .  .  .  operating  concepts  to  deliver  efficient  and  effective  perfonnance  to  the 
operating  forces”  (Wolfowitz,  2004).  Modeling  and  simulation  (M&S)  creates  an 
opportunity  to  explore  improvement  opportunities.  Applying  M&S  to  TLCM  efforts  has 
been  hampered  in  the  past  by  an  inefficient,  error-prone,  and  laborious  process  of  moving 
gathered  data  to  an  M&S  platform. 

TLCM  is  a  complicated  process.  The  tools  of  the  TLCM  facilitator  are  data 
collected  on  the  assets  that  are  being  managed  and  simulation.  The  data  collected 
provide  information  about  the  weapon  system  or  end  item.  Simulation  provides  a  way  to 
test  changes  to  the  system  before  costs  are  incurred.  The  use  of  data  collected  in 
simulation,  in  a  systematic  way,  allows  not  only  for  end  item  improvements,  but  also 
identifies  important  data  to  improve  models.  This  chapter  reviews  the  Marine  Corps’ 
TLCM  assets  and  the  focus  of  the  research. 

A.  TLCM  PROMOTES  ASSET  AVAILABLITY 

The  ability  of  the  Marine  Corps  to  accomplish  its  mission  is  reliant  on  personnel 
and  equipment  readiness.  TLCM  is  the  process  by  which  program  managers  (PMs) 
assess  a  principal  end  item  throughout  its  lifetime  and  ensure  its  availability.  Policy, 
procedural,  and  perfonnance  upgrades  may  be  required  to  ensure  the  most  efficient  use  of 
a  piece  of  equipment.  Through  M&S,  the  guesswork  may  be  taken  out  of  policy  and 
management  decisions.  This  research  bridges  a  gap  between  summarized  data  and  a 
modeling  platfonn,  to  allow  for  validation  of  the  model  build  process  and  improved 
TLCM  through  the  use  of  simulation. 

B.  RELATED  ELEMENTS  CENTRAL  TO  TLCM 

According  to  Marine  Corps  Order  (MCO)  4000.57,  TLCM  is  the  “formal  process 
to  identify,  analyze,  and  implement  synergistic  ‘cradle  to  grave’  solutions  that  optimize 
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the  acquisition/logistics  chain  across  the  Marine  Corps  in  support  of  operating  forces” 
(Kelly,  2005).  Included  in  TLCM  is  prognostics-  and  perfonnance-based  logistics.  To 
improve  the  maintenance  process  and  TLCM  in  general,  data  must  be  collected  and 
analyzed,  and  action,  perhaps  in  the  fonn  of  policy,  taken  to  improve  those  areas 
detennined  to  be  in  need  of  change. 

A  key  to  success  is  the  ability  to  collect  and  summarize  data,  and  then  use  this 
data  to  simulate  possible  realities.  Once  such  a  model  is  established,  factors  may  be 
adjusted  to  reflect  potential  management  decisions  or  policy  changes  aimed  at  improving 
performance,  increasing  availability,  or  lowering  the  cost  of  maintenance.  The  Marine 
Corps  has  contracted  to  develop  a  data  summarizing  tool  aimed  at  assessing  the  overall 
perfonnance  of  equipment.  Additionally,  a  modeling  platform  has  been  purchased  to 
build  predictive  models  that  provide  analysis  using  Monte  Carlo-based  simulation. 

TLCM  is  a  process.  Initially,  a  way  of  doing  business  is  established.  As  the  end 
item  and  its  supply  and  maintenance  systems  are  executed,  problems  resulting  from 
unforeseen  circumstances  are  identified.  With  the  use  of  M&S,  possible  system 
improvements  can  be  tested  prior  to  implementation.  In  the  case  of  TLCM,  there  are 
systems/processes  already  defined.  To  effectively  incorporate  modeling,  a  current  TLCM 
cycle  must  be  acknowledged  and  applied  within  the  model.  Figure  2  shows  how 
modeling  and  simulation  can  be  incorporated  in  the  existing  TLCM  process. 
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What  is  subject  to 
TLCM? 


Observe 

■System  or  Process - 


Figure  2.  The  incorporation  of  modeling  and  simulation  in  the  existing 

TLCM  process. 


1.  Collecting  and  Summarizing  Data 

To  affect  TLCM,  the  Marine  Corps  must  collect  data  and  articulate  it  in  some 
usable  format.  The  collection  and  summarizing  of  data  allows  the  system’s  operational 
effectiveness  to  be  reported,  given  its  current  and  previous  environment.  Effectiveness 
here  is  a  combination  of  the  system’s  availability,  reliability,  and  maintenance  operations 
costs.  Environmental  factors  may  include  operational  tempo  as  well  as  supply  and 
maintenance  systems  dynamics.  Examining  how  the  system  performs,  given  the  dynamic 
environment  in  which  the  system  is  running,  allows  the  analyst  to  determine  failure  rates, 
averages  of  shipping  times,  maintenance  evolutions,  and  other  factors  that  affect  the 
system’s  operational  availability. 


2.  Developing  Logical  Predictions  Pertaining  to  the  Impact  of 
Policy  Decisions 

The  effort  of  developing  a  system’s  operational  effectiveness  gives  the  analyst  an 
opportunity  to  collect  data  that  can  be  applied  to  M&S  efforts.  With  a  verified  and 
possibly  validated  model,  developed  policy  decisions  may  be  exercised  in  a  simulated 
environment  to  determine  their  overall  impact  on  defined  measures  of  effectiveness. 
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Through  simulation,  the  data  collected  directly  impacts  possible  futures.  Measures  of 
effectiveness  can  span  from  availability  of  the  end  item,  based  on  operational  tempo 
change,  to  the  impact  of  component  upgrades.  The  benefit  gained  from  building  verified 
and/or  validated  models  is  the  ability  to  quantify  the  benefits  of  program  changes  to  make 
more  informed  decisions. 

C.  SYSTEM  OPERATIONAL  EFFECTIVENESS  DECISION  SUPPORT 

TOOL  (SOE  DST)  AND  TLCM  ASSESSMENT  TOOL  (TLCM-AT)  FOR 

TLCM  WITHIN  THE  MARINE  CORPS 

Recognizing  the  importance  of  TLCM  in  support  of  the  warfighter,  the 
Marine  Corps  is  committed  to  the  development  of  TLCM  tools  and  processes.  The 
baseline  requirement  is  data  collection,  followed  by  model  development. 

1.  SOE  DST 

Marine  Corps  Systems  Command  (MCSC)  has  contracted  to  develop  the 
Web-based  SOE  DST  as  a  way  to  monitor  and  identify  areas  of  concern  for  maintenance 
and  supply  issues  (Alionscience,  2005).  The  SOE  DST  summarizes  historical  data  to 
evaluate  performance  and  develop  an  understanding  of  possible  future  requirements. 
Through  observation,  historical  maintenance  issues  may  shed  light  on  what  causes  a 
principal  end  item  (PEI)  to  be  in  a  nonavailable  state.  Perhaps  more  importantly,  SOE 
DST  produces  like  data  for  all  PEI’s  by  applying  algorithms  to  information  that  is 
gathered  from  a  myriad  of  sources.  This  data,  therefore,  is  universal  for  select  table  of 
authorized  materiel  control  numbers  (TAMCNs).  The  use  of  the  data  in  an  M&S 
environment  may  then  be  validated  across  TAMCNs. 

This  Web-based  application  provides  a  logical  source  for  data  to  be  used  in 
simulation  efforts.  A  model  must  be  verified  and,  eventually,  may  be  validated.  To 
reach  a  point  where  validation  is  possible,  the  result  produced  by  the  model  must  be 
measured  against  reality.  When  validating  a  model,  its  data  must  also  be  validated.  To 
reduce  variability  between  models  and  to  pennit  validation,  the  source  of  inputs  should 
be  standardized.  The  SOE  DST  produces  standardized  data.  Further,  the  use  of  the 
SOE  DST  allows  the  model-build  process  to  be  reusable  across  equipment  types. 
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2. 


TLCM-AT 


MCSC  has  established  a  contract  with  the  company  Clockwork  Solutions  and  has 
procured  TLCM-AT  as  a  closed-loop  simulation  model  (Clockwork  Solutions,  Inc, 
2007).  The  model  is  built  by  populating  41  separate  Access  database  tables.  The 
predictive  capacity  of  the  model  is  directly  related  to  the  inputs. 

TLCM-AT  uses  a  variety  of  input  data  and  provides  an  equal  amount  of  outputs  at 
simulation  end.  These  input  tables  represent  base  and  supply  structures,  starting  status  of 
equipment,  and  maintenance  factors.  The  simulation  takes  user  inputs  and  builds  a  fleet 
of  vehicles,  based  on  the  performance  and  engineering  specifications  of  PEIs.  With  these 
inputs,  TLCM-AT  logic  operates  the  modeled  equipment  as  specified,  subject  to  defined 
failure  rates  and  supply/maintenance  conditions.  Through  the  execution  of  the  model, 
multiple  results  are  gathered  and  summarized  as  output. 

Key  outputs  from  TLCM-AT  are  availability,  achieved  operating  hours,  time 
awaiting  maintenance  or  parts,  number  of  tasks  performed,  number  of  parts 
condemned/requested,  and  life  cycle  costs.  These  metrics  are  typical  questions  posed 
during  the  maintenance  process  evaluation  and  are  helpful  when  conducting  TLCM.  The 
simulation  allows  changes  in  processes  or  procedures  to  be  evaluated  prior  to 
implementation.  While  it  is  not  appropriate  to  take  the  results  as  the  absolute  solution, 
the  key  insights  derived  do  well  as  a  tool  to  help  decision  makers  evaluate  potential 
policy  decisions. 

D.  JOINT  EMPLOYMENT  OF  SOE  DST  AND  TLCM-AT 

The  full  potential  of  SOE  DST  and  TLCM-AT  requires  their  joint  employment. 
SOE  DST  collects  inputs  from  a  myriad  of  sources  and  systems  used  throughout  the 
Marine  Corps’  maintenance  and  supply  communities  to  report  historical  facts.  SOE  DST 
is  not  designed  to  act  as  a  predictive  tool.  TLCM-AT  has  the  ability  to  make  predictions 
based  on  projected  operational  tempo,  distributions  of  failure  rates,  and  repair  times,  but 
requires  current  data  in  order  to  produce  accurate  and  relevant  output.  While  one  system 
contains  infonnation  necessary  to  populate  the  other,  there  is  no  established  interface 
between  the  two  systems. 
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In  creating  models  using  historical  data  that  is  systematically  gathered  and 
synthesized,  the  model-build  process  can  be  validated  and  reused  across  equipment 
platforms.  With  a  clear  link  between  data  and  model,  there  is  a  reduction  in  variability  as 
a  result  of  subject  matter  expert  (SME)  opinion.  Additionally,  all  modelers  will  have 
access  to  the  same  data.  As  a  result,  the  quality  of  the  data  can  begin  to  be  measured. 

It  is  essential  to  have  a  tool  that  can  link  SOE  DST  and  TLCM-AT.  The  result  of 
the  link  is  an  opportunity  to  rapidly  employ  a  predictive  model  with  data  that  is  readily 
available  to  PMs.  If  this  model-build  process  is  validated,  it  may  be  used  to  guide  the 
decision-making  processes  pertaining  to  new  platfonn  purchases  or  upgrades,  potential 
policy  changes,  and  the  potential  impact  of  increased  operational  tempo.  If  the  model- 
build  process  cannot  be  validated,  it  will  serve  to  demonstrate  gaps  between  the  data 
collected  and  the  predictive  models  the  Marine  Corps  would  like  to  employ.  Regardless 
of  how  accurate  the  simulation  is,  it  can  potentially  provide  useful  insight  to  aid  decision 
makers. 

E.  PROBLEM  FORMULATION  AND  RESEARCH  DEVELOPMENT 

The  use  of  current  Web-based  applications  to  build  models  in  the  TLCM-AT 
environment  is  critical  to  the  future  successful  employment  of  the  two  tools.  The 
following  questions  guide  this  research: 

•  Can  the  development  of  TLCM-AT  models  be  aided  using  Visual  Basic 
for  Applications  (VBA)  and/or  other  programming  languages? 

•  What  gaps  are  there  between  data  summarized  in  SOE  DST  and  data 
required  in  TLCM-AT? 

•  Once  a  model  is  built  using  a  VBA  interface,  how  well  does  the  model 
assess  against  reality,  using  failures  as  the  measure  of  effectiveness 
(MOE)? 

•  Given  an  assessed  model  and  known  fluctuations  in  operational  tempo  or 
vehicle  population,  can  the  model  predict  changes  in  parts  failure  events? 

1.  Problem  Formulation 

The  TLCM  M&S  process  involves  data  gathering,  data  summary/analysis,  and 
model  development  and  discovery.  As  a  result,  many  agencies  are  required  to  fully 


6 


develop  a  problem  statement  and  subsequent  measures  of  effectiveness.  For  this 
research,  the  talents  of  many  professionals  were  gathered  and  melded  to  produce  a 
universally  agreed-upon  roadmap. 

The  International  Data  Farming  Workshop  (IDFW)  18  was  held  in 
Monterey,  California,  in  March  2009,  and  presented  an  opportunity  to  elicit  insight  from 
the  attending  TLCM  professionals.  A  complete  report  of  the  conference  will  be  released 
in  the  future  at  http://harvest.nps.edu/IDFW/18/idfwl8.html.  Through  IDFW  18,  this 
research  drew  from  the  experience  and  collaborative  efforts  of  many  members 
representing  the  multiple  layers  of  TLCM.  MCSC  and  Fleadquarters  Marine  Corps 
Installation  and  Logistics  (HQMC  I&L)  are  major  stakeholders  in  the  Marine  Corps’ 
TLCM  effort.  During  IDFW  18,  MCSC  hosted  a  group  focused  on  exploring  the  data-to- 
summary-to-model  process. 

The  IDFW  team  was  made  up  of  a  breadth  of  Marine  Corps  TLCM  professionals. 
As  part  of  the  group,  Dave  Sada  from  Andromeda  Systems  represented  the 
Marine  Corps’  data-gathering  element.  Alion  is  the  contract  holder  for  SOE  DST  and 
was  represented  in  the  group  by  Andy  Foote.  Clockwork  Solutions  and  Concurrent 
Technologies  Corporation  (CTC)  are  modeling  platform  developers  for  TLCM;  both  had 
modeling  professionals  attending  the  workshop.  Academic  professionals  from  the  Naval 
Postgraduate  School  also  participated.  Finally,  the  Joint  Light  Tactical  Vehicle  (JLTV) 
modeling  and  simulation  lead  from  the  Program  Executive  Office  for  Land  Systems 
attended  as  an  interested  customer  of  TLCM.  This  research  was  dramatically  aided  by 
this  gathering  of  stakeholders  in  the  process.  By  having  all  the  interested  parties 
participate  in  a  joint  effort,  a  universally  accepted  process  and  MOE  was  determined. 

The  process  for  the  research  was  to  establish  an  automated  link  between  the  data 
and  the  model.  Once  the  link  was  established,  the  models  were  then  assessed  using  root 
mean  squared  difference  (RMSD)  as  the  MOE  based  on  the  opinions  of  the  IDFW 
workshop  and  the  research  sponsor.  Once  a  model  was  assessed  as  “good,”  it  was  then 
exercised  as  a  predictive  tool.  Finally,  the  model’s  sensitivity  to  input  factors  was  tested 
using  Nearly  Orthogonal  Latin  Hypercube  (NOLH)  Design  of  Experiments  (DOE). 
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2. 


Link  Automation 


This  research  develops  a  Bridging  Operational  Logistics  Tool  (B-OLT)-an 
automated  link  between  SOE  DST  and  TLCM-AT.  B-OLT  is  created  using  VBA  in  the 
Excel  environment.  Housed  in  Excel,  B-OLT  is  transferable  within  the  Navy  Marine 
Corps  Intranet  (NMCI)  network.  B-OLT  may  be  easily  expanded  to  draw  input  from 
other  databases.  Additionally,  the  concept  of  a  single  source  document  that  houses  the 
data  necessary  to  build  a  predictive  model  may  be  expanded  to  other  modeling  platforms. 

In  this  research,  B-OLT  is  limited  to  SOE  DST  as  the  single  data  source  and  a 
basic  TLCM-AT  model.  Therefore,  the  findings  are  not  all  inclusive  and  further  research 
and  discovery  is  required.  The  opportunities  to  expand  the  research  are  numerous. 

3.  Assess  the  Goodness  of  the  Automated  Model-Build  Process 

The  models  built  using  the  automated  process  are  assessed  based  on  RMSD  of 
modeled  and  actual  parts  failures.  This  goodness  of  fit  is  appropriate  for  a  PM  and  their 
pursuit  of  TLCM.  Once  a  model  is  assessed  as  reflective  of  the  actual  failures  that  fed 
the  model,  it  may  be  exercised  to  acquire  answers  to  “what  if’  questions. 

For  this  research,  individual  models  are  built  and  the  raw  results  used  to  assess  the 
models.  These  results  are  presented  in  a  summary  of  all  the  models  as  well  as  individual 
TAMCN  results.  This  step  may  be  thought  of  as  a  demonstration  that  a  TLCM-AT 
model  can  be  automatically  built  and  produce  reasonable  results. 

4.  Initial  Examination  of  Predictive  Capacity 

Finally,  the  automated  model-build  process  is  used  as  a  predictive  tool.  It  is 
known  that  I  Marine  Expeditionary  Force  (I  MEF)  experienced  an  increase  in  vehicle 
population  and  operational  tempo  between  2002  and  2003.  The  SOE  DST  data  from 
2002  was  used  to  build  a  TLCM-AT  model.  Once  the  model  was  built,  the  operational 
tempo  and  vehicle  counts  were  adjusted  to  reflect  the  increase  in  these  two  areas  that 
would  have  been  known  by  I  MEF.  This  model  was  run  for  one  year  (2003)  and  the 
results  compared  to  actual  2003  SOE  DST  data.  The  initial  results  are  promising.  The 
automated  model  demonstrates  some  success  using  historical  data,  coupled  with  expected 
future  changes  to  make  predictions. 
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5.  Test  Sensitivities  by  Varying  Input  Factors 

Once  the  automated  model-build  process  is  assessed,  the  logical  next  step  is  to 
examine  sensitivities  of  the  models  to  the  factors  obtained  by  SMEs.  The  limitation  of 
data  to  a  single  SOE  DST  source  meant  that  five  factors  that  influence  failures  were  left 
to  other  data  sources.  It  is  important  to  detennine  how  much  of  the  model’s  variation  is  a 
result  of  these  factors  not  resident  in  SOE  DST,  in  order  to  determine  the  possibility  of 
improving  the  SOE  DST  platfonn.  Upon  completion  of  an  extensive  DOE,  it  was 
realized  that  a  reasonable  portion  of  the  variance  for  parts  failed  was  captured  in  the 
automated  model. 

In  this  research,  a  DOE  is  conducted  and  factors  of  interest  are  discovered.  In 
reality,  models  would  be  assessed,  adjusted,  and  then  used  to  answer  “what  if’  questions. 
As  this  was  not  the  purpose  of  the  research,  the  process  is  limited  to  discovery  alone  in 
order  to  demonstrate  the  concept. 

F.  BENEFITS  OF  AUTOMATING  THE  MODEL-BUILD  PROCESS 

This  research  provides  a  practical  interface  between  two  computer-based 
platforms  used  by  the  Marine  Corps  for  TLCM.  Once  the  link  between  data  gathered  and 
models  built  is  established,  the  models  are  then  verified  and  the  first  steps  toward 
validation  are  taken.  In  so  doing,  this  research  demonstrates  the  ability  to  link  the  two 
systems  in  a  manner  that  makes  modeling  and  simulation  readily  accessible  to  all  PMs  in 
the  Marine  Corps.  Automating  the  model-build  process  allows  models  to  be  built 
rapidly,  be  assessed,  and  then  used  for  “what  if’  analysis.  Using  standardized  inputs 
from  SOE  DST  means  the  process  may  be  shared  across  TAMCNs. 

G.  THESIS  ROAD  MAP 

The  following  chapters  provide  the  reader  with  a  brief  history  and  possible  future 

of  TLCM  within  the  Marine  Corps.  Chapter  II  focuses  on  the  SOE  DST  used  by  the 

Marine  Corps.  Next,  Chapter  III  explains  TLCM-AT  and  its  use  of  stochastic  modeling. 

The  TLCM-AT  modeling  structure  demonstrates  the  complexity  of  the  model  and  the 

dynamic  answers  possible  with  its  effective  employment.  A  bridge  between  the  data  the 

Marine  Corps  currently  collects  and  the  TLCM-AT  modeling  platform  is  developed  in 
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this  research.  Chapter  IV  describes  the  methodology  for  developing  the  code  behind  this 
work  and  some  of  the  limitations.  Once  the  model  is  developed  using  automation,  it  must 
be  assessed.  The  assessment  and  a  step  toward  verification  are  described  in  Chapter  V. 
With  modeling  and  simulation  comes  the  natural  question  of  “what  data  is  important?” 
Chapter  VI  demonstrates  the  use  of  model-building  automation  in  concert  with  DOE. 
Finally,  in  Chapter  VII,  the  insights  gained  through  this  research  are  discussed,  along 
with  future  research  opportunities. 
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II.  SOE  DST  FOR  DATA  SYNTHESIS  AND  AWARENESS 


SOE  DST  is  designed  to  synthesize  and  present  data  in  an  informative  manner. 
The  goal  is  to  facilitate  trend  analysis  and  assess  the  current  and  historical  availability 
posture  for  Marine  Corps  systems.  The  method  is  data  analysis  on  the  maintenance 
records,  with  an  understanding  that  the  records  are  often  inaccurate  or  incomplete. 

A.  PURPOSE  OF  SOE  WITHIN  LIFE  CYCLE  MANAGEMENT 

The  Department  of  the  Navy’s  Instruction  5400.15  series  defines  life  cycle 
management  as  “management  responsibility  for  a  program  that  encompasses  the 
acquisition  program,  in-service  support,  and  final  disposal”  (Winter,  2007).  DoD  policy 
is  that  a  system’s  PM  be  responsible  not  only  for  the  acquisition  of  a  system,  but  also  to 
remain  accountable  for  the  sustainment  of  the  system  over  its  lifetime.  As  directed  by 
MCSC’s  Strategic  Plan  2005-2009,  this  requires  PMs  to  monitor  and  improve  SOE. 
Measures  of  effectiveness  include  system  performance,  operational  availability,  process 
efficiency,  and  total  ownership  costs. 

The  monitoring  of  the  systems  is  handled  through  SOE  DST  Web-based  database, 
which  collects  data  from  a  myriad  of  reporting  sources,  as  illustrated  in  Figure  3.  With 
this  data,  PMs  can  identify  areas  that  require  improvement  and,  through  the  use  of 
simulation,  explore  potential  courses  of  action  in  order  to  make  decisions  to 
improve  SOE. 
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Figure  3.  System  Operational  Effectiveness  data-gathering  structure. 


B.  SOE  PROVIDES  A  PICTURE  OF  CURRENT  POSTURE 

Goal  three,  objective  one,  of  the  MCSC  2005-2009  Strategic  Plan  is  to  acquire  the 
capability  to  monitor  and  improve  SOE  throughout  the  life  cycle  of  systems  and 
equipment  (Catto,  2005).  The  mission  of  the  Capabilities  Assessment  Support  Center 
(CASC)  is  to  serve  as  the  focal  point  for  readiness,  reporting,  and  total  life  cycle  systems 
management  assessments  by  measuring  all  performance  aspects  of  fielded  Marine  Corps 
ground  equipment  throughout  the  life  cycle. 

To  appropriately  manage  performance,  and  to  answer  the  first  objective  of  goal 
three,  CASC  contracted  for  the  development  of  the  Web-based  SOE  DST  that 
summarizes  and  presents  data  to  help  PMs  evaluate  availability  and  define  potential  areas 
of  improvement.  This  Web-based  tool  captures  inputs  from  12  data  sources  (Figure  3) 
and  summarizes  that  data  into  trend  attributes  such  as  availability,  reliability, 
maintainability,  supportability,  and  total  ownership  costs.  As  data  is  gathered,  it  may 
then  be  used  to  highlight  areas  of  concern,  develop  courses  of  action  to  address  those 
concerns,  and  then  modeled  to  help  decide  which  is  the  most  effective. 
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C.  SOE  DST  HISTORICAL  RECORD  OF  SYSTEMS  MAINTENANCE 

SOE  DST  provides  a  historical  record  of  a  system  and  all  the  maintenance  and 
supply-related  transactions  associated  to  that  piece  of  equipment.  The  records  are  broken 
down  by  physical  location,  part,  or  single  piece  of  equipment,  depending  on  the  user’s 
request.  The  data  is  presented  in  the  same  repeatable  format,  no  matter  the  TAMCN  or 
dates  requested. 

1.  Display  and  Synthesis  of  Raw  Data 

The  “PartsUsage.xls”  file  from  SOE  presents  a  single  TAMCN’s  failed  parts  for  a 
given  time  period,  in  a  given  location.  This  Excel  file  has  13  columns  of  data  for  each 
part  that  failed  in  a  given  time  frame,  for  a  given  piece  of  equipment  (taken  from  the 
SOE  DST  help  file). 

•  National  Stock  Number  (NSN):  Unique  numerical  identifier  for  each 
part  of  the  selected  equipment. 

•  Part  Name:  Supported  Activities  Supply  System/Federal  Logistics 
Record  (SASSY/FEDLOG)  text  description  of  the  part. 

•  Part  Count:  Required  number  of  a  given  NSN  for  selected  date  range. 
This  quantity  is  bounded  by  the  Stock  List  (SL)-Quantity  of  that  part. 

•  Unit  Price:  Current  SASSY/FEDLOG  part  cost  for  the  part. 

•  Order  Count:  Number  of  given  part  ordered  during  the  data  range.  This 
quantity  is  NOT  bounded  by  the  SL-Quantity. 

•  Extended  Price:  Total  cost  of  NSN  ordered. 

•  Equipment  Repair  Order  (ERO)  Count:  Number  of  EROs  that  NSN 
was  required  during  the  selected  date  range. 

•  Average  Logistics  Response  Time  (LRT):  Mean  of  the  LRT  (number  of 
days  between  date  that  part  is  ordered  and  date  it  is  received)  for  a  given 
NSN  during  selected  date  range. 

•  Failure  Rate:  Measure  of  reliability,  in  failures  per  million  calendar 
days,  for  given  NSN  during  selected  date  range.  The  failures  per  million 
calendar  days  figure  is  converted  to  failures  per  calendar  day  when  the 
PartsUsage  file  is  downloaded  from  the  Website. 

•  Percentage  of  Weapons  Systems  (WS)  Replaced:  Measure  of 
percentage  of  weapon  systems  (WS)  that  given  NSN  was  replaced  during 
selected  date  range. 
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•  Stock  List  Quantity:  Identifies  total  number  of  given  NSN  that  are 
required  on  selected  weapon  system. 

•  Criticality  Code:  The  criticality  code  assigned  to  the  NSN. 

•  Source,  Maintenance,  and  Recoverability  (SMR)  Code:  Code 

associated  with  the  NSN  that  is  used  to  determine  the  echelon  of 
maintenance  authorized  to  condemn,  repair,  or  remove  and  item. 

During  IDFW  18,  it  was  agreed  that  the  data  collected  at  various  points  in  the 
maintenance  and  supply  chains  can  be  flawed  with  user  error.  Some  of  the  issues  include 
missing/incorrect  serial  numbers  for  equipment,  incorrect  order  quantities,  missing  SMR 
codes,  and  a  host  of  others.  An  important  benefit  of  SOE  DST  to  TLCM  is  the  filtering 
done  behind  the  scenes  to  make  up  for  gaps  in  the  data.  Once  the  gaps  are  filled, 
averages  are  presented  to  the  user  for  various  metrics.  For  the  purpose  of  this  research, 
the  most  important  metric  is  the  failure  rate. 

2.  Failure  Rates  and  the  Stochastic  Modeling  Process 

Probabilistic  and  detenninistic  are  two  common  approaches  to  modeling  and 
simulation.  Detenninistic  models  do  not  utilize  random  variables,  and  are  typically  more 
appropriate  for  use  in  clearly  defined  and  unchanging  cause-and-effect  relationships. 
Scheduling  of  aircraft  may  be  an  example  of  an  appropriate  deterministic  model 
approach,  in  that  missions,  costs,  and  benefits  are  clearly  defined  and  the  goal  is  to  decide 
the  optimal  combination  of  scheduling  factors.  A  probabilistic  model  can  capture  the 
random  nature  inherent  in  many  logistics  systems. 

In  probabilistic  models,  historical  reference  may  be  used  to  define  probabilities 
after  distributions  are  fitted  to  the  data.  Once  a  model  is  built  in  this  manner,  it  allows  an 
analyst  to  take  advantage  of  historical  reference  in  order  to  develop  an  understanding  of 
possible  futures  to  aid  in  decision-making  processes.  To  ensure  that  the  possible  futures 
are  believable,  validation  of  the  model  is  necessary  and  development  of  confidence 
intervals  is  appropriate.  The  keystone  to  a  probabilistic  model  is  capturing  the  right 
distributions  for  inputs  within  the  model. 

SOE  DST  provides  an  average  that  may  be  applied  to  a  probabilistic  model  if  the 
model  uses  a  defined  distribution,  such  as  the  exponential  distribution.  Though  with 
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equipment  failures  it  is  typical  to  use  a  Weibull  distribution,  to  get  a  less  variable  picture 
of  potential  failures,  the  exponential  is  sometimes  used  (Devore,  2008)  as  a  substitute. 
The  Weibull  distribution  allows  for  infant  mortality  or  wear-out  mortality  depending  on  a 
particular  part’s  tendency.  Since  SOE  DST  provides  an  average,  the  Excel-based  file 
may  easily  be  transferred  into  inputs  into  a  probabilistic  model  if  an  exponential 
distribution  is  assumed.  Because  SOE  DST  summarizes  the  data  from  a  larger  database, 
a  user  of  B-OLT  cannot  derive  the  shape  parameter  required  to  use  a  Weibull 
distribution.  This  may  or  may  not  be  a  factor.  For  example,  when  applied  to  electronic 
TAMCNs,  it  may  be  detennined  that  a  Wiebull  is  required.  When  working  with  the 
assembled  team  at  IDFW  18,  it  was  agreed  that,  given  the  nature  of  the  legacy  JLTV 
vehicle,  using  an  exponential  distribution  with  analyzed  data  was  more  desirable  than 
attempting  to  reanalyze  the  same  data  in  order  to  use  a  Weibull.  Future  SOE  DST 
methodologies  may  aide  modeling  if  shape  parameters  are  computed. 

The  benefit  to  using  synthesized  data  from  a  common  repeatable  source,  such  as 
SOE  DST  in  the  modeling  process,  is  that  the  process  of  summarizing  the  data  is 
universal  across  TAMCNs.  This  means  that  a  model  that  effectively  uses  the  SOE  DST 
data  for  one  TAMCN  will  use  the  SOE  DST  data  from  a  different  TAMCN  in  the  same 
manner,  with  similar  results.  A  model-build  process,  in  turn,  that  systematically  takes  the 
results  of  SOE  DST  and  translates  it  into  a  model  may  be  reused  across  multiple 
TAMCNS.  Further,  if  the  models  are  built  with  the  same  process,  using  the  same  data 
source,  and  are  repeated  over  varying  conditions  and  produces  statistically  similar 
predictive  results,  the  entire  modeling  process  may  be  validated  over  time. 
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III.  TLCM-AT:  A  STOCHASTIC  MODELING  PLATFORM 
DESIGNED  TO  USE  CURRENT  DATA  FOR 
QUANTITATIVE  PREDICTIONS 


TLCM-AT  is  a  stochastic  model  platform.  The  building  of  models  within  the 
TLCM-AT  environment  relies  on  data  that  has  been  collected,  cleaned,  and  analyzed. 
One  purpose  of  models  built  in  the  TLCM-AT  environment  is  to  conduct  “what  if’ 
analysis  designed  to  gain  insight  on  the  impact  of  potential  decisions.  This  chapter 
discusses  TLCM-AT  methodology  and  processes. 

A.  TLCM-AT:  A  MODEL  PLATFORM,  NOT  A  MODEL 

The  Marine  Corps,  as  part  of  goal  three,  objective  one  of  the  MCSC  2005-2009 
Strategic  Plan,  requires  the  capability  to  use  collected  historical  data  to  create  models 
useful  in  “what  if’  analysis.  Clockwork  Solutions  provided  a  tool  that  was  originally 
developed  for  aircraft  maintenance.  The  sophisticated  platfonn  takes  user  inputs  through 
multiple  interfaces  and  allows  the  user  to  detennine  the  effects  of  policy  and  management 
decisions  before  implementing  a  change  in  the  real  world.  This  use  of  simulation  reduces 
the  necessity  of  performing  costly  trial-and-error  testing.  TLCM-AT  was  developed  to 
assist  weapon  systems  fleet  managers  with  evaluating,  quantifying,  and  reducing  life 
cycle  costs,  without  adversely  impacting  fleet  readiness  and  availability  (Clockwork 
Solutions,  2005).  The  continuous-loop  representation  of  the  life  cycle  of  any  weapons 
system  combines  operations,  maintenance,  and  logistics,  as  shown  in  Figure  4.  The  blue 
(inner  loop)  portions  are  input  variables,  while  the  black  (outer  loop)  portions  are  outputs 
from  the  model.  The  level  of  fidelity  available  with  the  TLCM-AT  model  is  greater  than 
that  of  the  outputs  from  SOE  DST,  which  results  in  some  challenges  during  the  model¬ 
building  that  are  discussed  in  Chapter  IV. 
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Figure  4.  The  TLCM-AT  continuous-loop  model  (From:  ATLAST  Technical 
Reference  Manual,  1992-2005).  The  outer  green  boxes  represent  model  modules.  Each 
inner  box  is  an  external  factor  that  may  effect  any  of  the  outer  modules.  The  inner  blue 
terms  represent  user  inputs  and  the  outer  black  terms  are  model  outputs.  This  is  best 

viewed  in  color. 

An  individual  weapons  system  TLCM-AT  model  is  composed  of  six  interactive 
components,  according  to  the  Clockwork  Solutions’  technical  manual  (1992-2005): 


•  Initialization:  The  initial  condition  and  location  of  systems  and  parts  in 
the  model;  parts  or  systems  in  maintenance  at  the  start  of  the  simulation. 
This  is  an  opportunity  to  define  the  age  of  the  fleet,  if  applicable/possible. 

•  System  Module:  Work  breakdown  structure  of  the  system  and  its 
variants;  the  fleet  disposition  including  acquisitions  and  redeployments; 
base  support  structure;  three  echelon  levels,  to  include  ship  times. 

•  Operations  Module:  Current  and  future  operations  according  to  base 
location,  platfonn  type,  or  serialized  system;  unscheduled  removal  rates 
and  life  limits. 
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•  Maintenance  Module:  Actions  on  a  component  after  it  has  entered 
maintenance;  capacity  constraints.  Maintenance  task  times,  logistics 
consequences,  and  not  repairable  this  station  (NRTS)  probabilities. 

•  Sustainment  Module:  Spares,  lateral  resupply,  depot  upgrades,  and 
induction  programs,  reprovisioning. 

•  Cost  Module:  The  cost  of  purchases  and  activities,  including  (but  not 
limited  to)  maintenance,  training,  initial  and  reprovisioning  of  parts, 
storage,  shipping,  and  upgrades.  There  are  nine  total  categories  of  cost. 

B.  TLCM-AT:  INPUT  CONSISTENCY  CONCERNS 

With  the  above  interactive  components  come  many  opportunities  to  adjust  how 
systems  are  modeled.  As  models  are  not  necessarily  a  reflection  of  reality,  but  always  a 
reflection  of  what  is  contained  in  the  model,  greater  accuracy  in  the  model  naturally 
translates  into  a  clearer  and  more  accurate  view  of  eventual  reality.  The  MCSC  strategic 
plan  objective  calls  for  a  link  between  the  data  that  is  summarized  in  SOE  DST  and  the 
model-build  process.  Removing  SME  inputs  in  the  foundation  of  the  model,  and  limiting 
the  inputs  to  a  single  data  source,  allows  for  repeatability  in  the  process,  which  is  critical 
to  future  validation  efforts. 

1.  SME  Inputs  are  Often  Used,  but  are  not  Universally  Consistent 

Prior  to  this  research,  TLCM-AT  models  relied  on  data  from  a  myriad  of  sources 
particular  to  the  modeler’s  span  of  influence  or  access.  As  a  result,  each  model  was  built, 
verified,  and  then  used  to  answer  the  questions  specific  to  the  time  the  model  was 
contracted.  Models  could  occasionally  be  reused  to  answer  emerging  questions,  but  not 
necessarily  as  a  matter  of  course. 

TLCM-AT  models  are  built  with  available  SME  inputs  and  summarized  data. 
Clockwork  Solutions  has  used  SOE  DST  data  in  the  past,  with  adopted  rules  for  handling 
gaps  and  supplementing  with  SME  input.  In  weapons  systems  maintenance,  SMEs 
develop  a  general  understanding  for  the  weapons  system.  As  they  may  be  grown  in  a 
particular  location  within  a  particular  operational  tempo,  they  develop  expertise  in  how 
that  weapon  system  functions  in  their  current  environment.  Finally,  the 
model-builder  may  not  have  access  to  the  best  SME.  These  factors  make  it  essential  to 
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augment  the  SME  in  the  model-build  process  wherever  possible.  This  augmentation 
should  come  in  the  fonn  of  objective  data. 

Through  B-OLT  automation,  future  efforts  may  be  applied  to  how  the  data  is 
gathered  and  summarized.  Additionally,  a  quantitative  analysis  may  be  applied  to 
improving  the  processes  by  which  the  Marine  Corps  collects,  stores,  and  displays  its  data. 

SMEs  are  critical  when  anticipating  what  may  happen  in  the  operational 
environment.  Part  of  the  research  was  to  test  the  predictive  nature  of  models  built.  To 
test  this  capability,  a  period  of  known  change  was  selected  and  modeled,  using  only  two 
pieces  of  infonnation. 

•  The  historical  maintenance  data  taken  from  SOE  DST  for  the  prior  year. 

•  The  information  that  would  have  been  known  by  an  SME  in  terms  of 
changes  in  operational  tempo  and  vehicle  populations. 

To  obtain  point  two,  a  modeler  requires  the  input  that  only  a  SME  can  provide. 
The  difference  in  this  case  is  that  the  SME  is  providing  input  particular  to  the  current 
emerging  situation.  The  historical  perfonnance  data  is  still  resident  in  a  common  picture, 
single  source  platfonn,  SOE  DST. 

2.  Historical  Data  Inputs 

TLCM-AT  is  an  Access-based  simulation  that  accepts  as  much,  or  as  little,  detail 
as  required  to  answer  the  modeler’s  question.  Prior  to  the  automation  of  user  inputs 
through  B-OLT,  developed  as  part  of  this  research,  historical  data  was  left  up  to  the  data 
sources  available  to  the  modeler.  With  SOE  DST  alone,  there  are  12  sources  for 
maintenance  and  supply  data.  Without  continuity  of  data  sources,  there  is  no  way  to 
replicate  models  built  between  modelers. 

The  SOE  DST  Parts  Usage  file  is  enough  to  build  a  base  model.  TLCM-AT 
models  have  a  minimum  requirement  for  the  modeler  to  identify  the  vehicle  platforms, 
the  parts  that  make  the  platform  (does  not  have  to  be  complete),  the  base  infrastructure, 
and  failure  rates.  The  models  built  can  also  be  given  shipping  times  between  bases,  rules 
pertaining  to  lateral  support,  and  a  host  of  other  optional  factors.  The  master  data 
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repository  (MDR)  that  houses  the  maintenance  and  supply  historical  data  for  the 
Marine  Corps  contains  a  majority  of  the  data  necessary  to  at  least  build  a  minimal 
TLCM-AT  model. 

C.  TLCM-AT  CAN  IMPROVE  MAINTENANCE  PROCESS 

UNDERSTANDING 

TLCM-AT,  or  other  simulations,  may  be  employed  to  explore  management  or 
policy  decisions  before  committing.  This  allows  decision  makers  to  gain  insight  to  help 
understand  the  full  consequences  of  that  decision.  Once  the  TLCM-AT  model  is  built 
and  assessed,  input  factors  may  be  changed  to  detennine  potential  improvements  in 
measures  of  effectiveness.  Models  may  also  be  used  to  determine  what  factors  may  help 
to  improve  specified  system  outputs. 

Models  that  are  shown  to  reflect  reality  provide  insight  into  possible  futures. 
When  decision  makers  are  considering  supply  positioning  options,  various  courses  of 
action  may  be  modeled  to  demonstrate  the  effect  on  PEI  availability,  for  example.  If  the 
possible  policy  demonstrates  a  reduction  in  availability  in  the  model,  it  is  reasonable  to 
assume  the  same  may  hold  for  the  real  world.  This  requires  that  the  model  demonstrate 
an  appropriate  level  of  similarity  to  the  real  world.  This  confidence  comes  from  repeated 
use  of  the  model,  or  the  model-build  process,  and  the  subsequent  accumulated  validation. 

Similarly,  an  assessed  model  may  be  used  to  detennine  where  policy  may  be 
improved.  Through  DOE,  factors  are  adjusted  in  order  to  demonstrate  which  are  the  most 
influential  in  explaining  the  variability  of  a  specified  measure  of  performance.  Again, 
prior  to  any  reliance  being  placed  in  the  model’s  insights,  there  must  be  some  verification 
of  its  perfonnance.  As  stated  by  the  DoD  Modeling  and  Simulation  Coordination  Office 
(MSCO),  “it  is  virtually  impossible  to  separately  evaluate  a  model  and  the  data  it  uses” 
(Modeling  and  Simulation  Coordination  Office,  2006). 

TLCM-AT  models  built  using  SOE  DST  data  are  based  in  a  standardized  format. 
Through  the  research,  it  was  determined  that  models  built  using  SOE  DST  data  acted 
similarly  across  vehicle  variants.  This  understanding  makes  it  clear  that  if  one  can  adapt 
a  process  of  model-building  and  get  expected  results,  then  the  process  itself  can  be 
verified.  Further  work  could,  in  fact,  lead  to  a  validation  of  the  model-build  processes, 

given  the  standard  data  source  and  the  model  platform. 
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IV.  B  OLT  MOVES  SOE  DATA  INTO  THE 
TLCM-AT  PLATFORM 


This  research  initially  focused  on  the  model-build  process.  Old  constructs  had 
TLCM-AT  models  built  and  “what  if’  analysis  conducted  by  Clockwork  Solutions. 
While  licenses  are  available  to  the  Marine  Corps,  the  model-build  process  is  currently  too 
complicated  to  be  functionally  practical.  However,  data  to  feed  the  model  is  available  to 
every  Marine.  SOE  DST  provides  a  single  document  that  satisfactorily  provides  most  of 
the  infonnation  needed  to  build  a  functional  TLCM-AT  model.  The  infonnation  not 
resident  in  SOE  DST  can  be  found  in  Visibility  &  Management  of  Operation  &  Support 
Cost  (VAMOSC)  and  SME  input.  This  research  uses  as  its  research  platform  the  legacy 
JLTV  or  High  Mobility  Multipurpose  Wheeled  Vehicle  (HMMWV). 

A.  SOE  DATA  CONVERSION  TO  TLCM-AT 

As  the  lexicon  for  SOE  DST  and  TLCM-AT  are  not  alike,  there  are  some 
assumptions  that  must  be  made  and  rules  established.  TLCM-AT  is  a  series  of  modules 
within  the  overall  modeling  platfonn.  The  system,  base,  and  maintenance  modules  are 
the  basic  modules  in  the  model,  and  cause  some  difficulty  when  converting  SOE  DST 
data  into  a  TLCM-AT  model. 

1.  Building  the  System  Module  within  TLCM-AT 

The  system  module  consists  of  input  data  that  represents  a  work  breakdown 
structure  of  the  system  and  its  variants;  the  disposition  of  the  fleet,  including  acquisitions 
and  redeployments;  and  the  three  echelons  of  maintenance  support  structure.  Currently, 
the  SOE  DST  fonnat  does  not  allow  for  an  indenturing  of  parts  within  the  vehicle,  nor 
does  it  define  parts  by  part  number.  As  a  result,  each  part  that  appears  on  the  Parts  Usage 
Report  in  SOE  DST  is  treated  as  a  line  replaceable  unit  (LRU).  For  TLCM-AT,  all 
events  begin  with  an  LRU  event  and  therefore  each  part  creates  tasks  on  the  event  list  at 
the  start  of  the  simulation.  Figure  5  displays  the  TLCM-AT  possible  breakdown  structure 
and  the  limited  structure  possible  using  the  SOE  DST  PartsUsage  file  alone. 
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Work  Breakdown  Structure  Possible 


Vehicle 


►  LRU  1 


►  LRU  2 


Work  Breakdown  Structure  using  SOE  DST 


Figure  5.  The  work  breakdown  structure  possible  in  TLCM-AT,  as  compared  to  the 
limited  structure  in  an  SOE  DST-driven  model. 

An  LRU-only  model  has  the  potential  to  cause  large  memory  storage 
requirements  with  the  model  run.  To  help  offset  the  storage  constraints  imposed  by  an 
LRU-only  model,  the  model  is  limited  to  a  single  vehicle  variant  and  a  single  base 
structure.  Figure  6  shows  the  base  structures  possible  in  TLCM-AT  and  the  base 
structure  used  to  accommodate  the  LRU-only  model  using  SOE  DST  data. 
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Base  Structure  Possible 


Base  Structure  using  SOE  DST 

Figure  6.  The  base  structure  possible  in  TLCM-AT,  as  compared  to  the  limited  base 

structure  in  an  SOE  DST-driven  model. 

The  LRU  only  model  limits  the  scope  of  future  “what  if’  analysis.  Using  the 
single  base  structure  the  model  cannot  exercise  lateral  resupply  or  variations  in 
intermediate/depot  level  supply  infrastructure.  Flaving  only  LRUs  identified,  the  model 
is  limited  to  the  size  of  the  fleet  it  can  model  based  on  memory  storage  requirements. 
This  unnecessarily  limits  the  dynamics  possible  within  the  TLCM-AT  environment.  As  a 
result,  future  improvements  must  include  indenturing  of  parts  and  the  incorporation  of  a 
parts  number  structure  within  the  SOE  DST. 

2.  Building  the  Operations  Module  within  TLCM-AT 

The  operations  module  consists  of  the  operational  profile,  unscheduled  removal 
rates  (URR),  and  life  limits.  The  fidelity  is  left  up  to  the  modeler.  In  the  case  of  the 
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operations  profile,  it  is  possible  to  define  usage  per  serialized  platform  or  maybe  as 
general  as  a  defined  amount  of  hours  per  platform  per  base.  Acceptable  units  in 
TLCM-AT  are  operating  hours  or  miles.  There  is  an  option  to  define  two  more  units, 
such  as  number  of  starts,  if  necessary.  In  the  SOE  DST-based  model,  there  is  currently 
no  reliable  way  to  capture  average  usage  rates  at  any  level.  This  is  a  function  of  poor 
data  collection  resulting  in  missing  figures  when  SOE  DST  conducts  its  analysis.  As  a 
result,  this  research  resorted  to  VAMOSC  figures  for  average  annual  miles.  In  the  future, 
it  may  be  appropriate  to  include  an  average  annual  usage  rate  within  the  PartsUsage  file 
of  SOE  DST. 

The  SOE  DST  equivalent  for  the  URR  is  the  failure  rate.  When  a  user  downloads 
the  PartsUsage  file  from  SOE  DST,  they  are  provided  a  failure  per  day  rate  for  each  part. 
For  TLCM-AT,  failures  are  treated  as  a  Weibull  distribution.  SOE  provides  an  average 
failure  rate  across  the  fleet.  Without  the  raw  data,  a  B-OLT  user  cannot  compute  the 
shape  parameter.  Without  knowing  a  shape  parameter,  SOE  DST-driven  TLCM-AT 
models  use  the  exponential  distribution  (i.e.,  we  assume  a  shape  parameter  value  of  1). 
Additionally,  TLCM-AT  works  with  failures  per  kilo-miles  and  the  SOE  DST  failures 
per  day  must  be  converted  using  the  following  formula: 

\  Failures  9\.25days  1  Qtr  Failures 

Day  Qtr  X  kilo -miles  kilo -miles 

Once  the  failures  per  kilo-mile  are  determined,  the  miles  per  hour  (mph)  must  be 
detennined  to  get  an  estimated  depiction  of  system-level  operational  usage.  An  SME 
opinion  of  the  average  speed  the  vehicles  travel  is  required,  as  SOE  DST  does  not 
currently  provide  this  infonnation.  In  garrison  (noncombat)  operations  it  is  presumed 
that  vehicles  drive  at  an  average  20  mph  rate.  In  combat,  this  figure  is  boosted  to 
35  mph.  With  the  VAMOSC-provided  average  miles  driven  and  the  SME-provided  mph 
estimates,  the  operations  module  may  be  created. 

3.  Building  the  Maintenance  Module  within  TLCM-AT 

The  maintenance  module  accepts  inputs  pertaining  to  logistics  consequences, 
maintenance  task  times,  and  NRTS  events.  NRTS  is  a  TLCM-AT-specific  acronym  that 

allows  the  model  to  address  cases  when  a  component  must  be  evacuated  to  the  next 
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higher  echelon  of  maintenance.  Typically,  with  Marine  Corps  maintenance,  this  is  tied 
directly  to  the  specific  part  requiring  maintenance. 

The  maintenance  cycle  begins  with  an  LRU  event  that  is  caused  by  the 
unexpected  removal  rates  and  life  limits  definitions  provided  in  the  operations  module. 
The  end  of  the  maintenance  cycle  depends  on  the  object  of  interest.  A  platform  is  out  of 
the  maintenance  cycle  when  all  of  its  slots  (defined  in  the  system  module)  are  filled  with 
operable  parts. 

When  there  is  an  LRU  event,  the  first  step  is  an  inspection  of  the  vehicle,  which 
always  occurs  in  the  operational  level.  The  inspection  results  in  four  possible  object 
statuses:  operational,  no-fault-found,  repairable  failure,  or  nonrepairable  failure.  Since 
the  maintenance  cycle  begins  with  an  LRU  event,  the  LRU  is  removed  from  the  platform 
and  inspected  first.  Based  on  the  possible  status,  the  following  will  result: 

•  Operational:  LRU  is  never  removed  and  the  platform  immediately  goes 
back  in  the  operation  cycle. 

•  No-Fault-Found:  LRU  removed,  inspected,  and  replaced  in  the  platform, 
as  if  it  were  detennined  to  be  operational. 

•  Repairable  Failure:  LRU  is  removed  and  replaced  in  the  platform  if 
there  is  a  spare  LRU  available.  The  LRU  enters  the  maintenance  cycle.  A 
module,  part,  or  subpart  is  detennined  to  be  damaged  and  that  module, 
part,  or  subpart  is  removed  from  the  LRU.  The  module,  part,  or  subpart  is 
entered  into  the  maintenance  cycle.  If  a  replacement  is  available,  the 
LRU’s  defective  part  is  replaced  and  LRU  is  put  back  into  the  platfonn  (if 
no  spare  LRU  was  previously  available)  or  into  shelf  stock. 

•  Nonrepairable  Failure:  Same  as  repairable  failure,  but  the  LRU  is 
discarded  and  new  part  installed. 

The  maintenance  cycle  from  LRU  event  to  repair  may  be  seen  in  Figure  7. 
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Figure  7.  The  maintenance  flow  module  in  TLCM-AT.  This  is  best  viewed  in  color. 


For  the  SOE  DST-based  model,  the  SMR  codes  are  used  to  define  logistics 
consequences.  Inspection  times  are  based  on  the  third  digit  of  the  SMR  code,  which, 
according  to  NAVSUP  7-19,  is  the  lowest  maintenance  level  authorized  to  remove, 
replace,  and  repair  the  part.  The  model  logic  allows  times  to  be  allocated  for  inspection, 
repair,  and  shipment.  The  Maintenance  Allocation  Chart  (MAC)  associated  with  every 
TAMCN  provides  expected  times  to  perform  maintenance  tasks.  For  the  legacy  JLTV, 
the  MAC  is  found  in  TM  9-2320-280-20-3.  Table  1  defines  the  high  and  low  times 
associated  to  inspections,  repairs,  and  tear  times,  based  on  which  level  of  maintenance  is 
allowed  to  repair  or  replace  the  item.  Currently,  there  is  no  distribution  associated  with 
these  times. 
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Table  1.  Inspection  time,  in  hours,  assumptions  applied  to  an  SOE  DST-driven  model. 


Action 

MAC 

Nomenclature 

Range 

1st  Echelon 

2nd  Echelon 

3rd  Echelon 

Inspection 

Inspection  and  Test 

Low 

0.0 

0.2 

0 

Avg 

0.18 

2.0 

0 

High 

0.7 

2.0 

0 

Repair 

Overhaul,  Repair 
and  Service 

Low 

0.0 

0.0 

0 

Avg 

0.63 

3.8 

0 

High 

3.5 

16.0 

0 

Tear 

Install  and  Replace 

Low 

0.0 

0.0 

0 

Avg 

1.1 

3.2 

0 

High 

10.0 

32.7 

0 

The  tear  times  are  used  to  define  how  long  it  takes  an  object  to  be  removed  from 
the  platfonn.  Since  B-OLT  creates  an  LRU-only  model,  and  there  is  no  flexibility  in 
defining  separate  removal  rates  based  on  indenture,  additional  time  had  to  be  added  to 
parts  that  are  typically  evacuated  to  a  higher  echelon  of  maintenance.  An  LRU  in  the 
TLCM-AT  logic  is  removed  at  the  operational  level.  If  it  is  NRTS,  then  it  is  evacuated  to 
the  next  higher  echelon  of  maintenance  and  the  modules  that  are  defective  within  the 
LRU  have  their  own  associated  intennediate-  or  depot-level  tear  times.  However,  given 
the  limits  of  an  SOE  DST-based  model,  the  shipping  times  had  to  be  added  to  the 
operational  tear  times.  It  is  important  to  remember  that  each  LRU  is  removed  at  the 
operational  level.  Table  2  demonstrates  the  time  added  to  tear  time  in  TLCM-AT  to 
account  for  shipping  the  part  or  PEI  to  the  appropriate  echelon  of  maintenance. 

Table  2.  Tear  time  assumptions  applied  to  an  SOE  DST-driven  model. 


Lowest  Remove/Replace 

Ship  Time  to  Repair  Level 

Single 

DOE  range 

Operational 

0 

0 

Intermediate 

5  days 

1-20  days 

Depot 

30  days 

20-60  days 

It  is  possible  for  each  part  to  define  maintenance  allocation  times  based  on  the 
maintenance  allocation  charts.  However,  the  legacy  JLTV  MAC  does  not  define  parts  by 
NSN  and  therefore  there  is  no  practical  way  to  merge  SOE  DST  data  with  MAC.  The 
MAC  nomenclature  does  not  match  the  nomenclature  used  in  SOE  DST  and  there  are 
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multiple  NSNs  that  may  or  may  not  be  the  same  part.  Without  a  clear  understanding  of 
the  indenture  structure  and  which  parts  require  how  much  time,  based  on  MACs,  the 
SOE  DST-based  model  is  limited  to  approximations  of  times,  based  on  echelon  of 
maintenance  categories. 

4.  Building  the  Sustainment  Module  within  TLCM-AT 

The  sustainment  module  allows  for  shipment  times,  spares  allocation,  lateral 
resupply,  preferred  buys,  and  a  depot  upgrades  program.  The  SOE  logistics  response 
time  is  the  average  ship  time  for  that  repair  part  and  can  be  used  as  shipping  times.  This 
is  only  true  if,  in  reality,  there  are  never  spares  on  hand.  For  the  model,  it  would  be  best 
to  fit  a  distribution  to  all  LRUs  (perhaps  separated  by  criticality  code)  and  use  this  as  the 
shipping  time.  Given  the  limitations  of  SOE  DST  and  an  interest  in  capturing  all  possible 
failures,  the  models  associated  with  this  research  flooded  the  supply  system  with  100 
parts  at  each  level  of  maintenance.  By  always  having  a  spare  part  in  the  system,  there 
was  no  doubt  that  the  modeled  vehicles  would  achieve  their  defined  operational  tempo 
and  would,  therefore,  break  as  often  as  possible. 

B.  B-OLT  CODING  IN  VISUAL  BASIC  FOR  APPLICATIONS  (VBA) 

B-OLT  is  established  in  an  Excel  environment  and  executes  using  VBA  code. 
The  driving  factor  behind  Excel  implementation  is  the  universal  acceptance  of  Excel  by 
potential  users.  B-OLT  can  be  placed  on  any  machine  in  the  NMCI  network  that  has  the 
TLCM-AT  platform  installed  and  run  without  requiring  the  user  to  learn  the  TLCM-AT 
application.  However,  assessment  of  the  model  built  must  be  done  prior  to  conducting 
any  “what  if’  analysis.  For  this  reason,  PMs  must  work  in  concert  with  modeling 
professionals  employing  any  simulation.  The  overall  place  for  B-OLT  is  between 
SOE  DST  and  TLCM-AT  to  bridge  the  gap  between  data  collected/summarized  and 
simulation  as  demonstrated  in  Figure  8. 
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Figure  8.  The  link  developed  with  B-OLT  between  data  and  simulation. 


B-OLT  links  the  SOE  DST  Excel  document  with  limited  user  inputs  to  the 
Access-based  TLCM-AT  platfonn.  The  user  interface  fills  gaps  in  the  SOE  DST 
PartsUsage  file  data.  Specifically,  the  user  must  provide  maintenance  times  (from 
MAC),  shipping  times  (from  SME),  annual  miles  (from  VAMOSC),  average  mph  (from 
SME),  and  minimum  dollar  value  of  interest  (from  SMEs).  The  minimum  dollar  value  of 
interest  is  aimed  at  helping  computational  time.  In  this  research,  the  parts  that  are 
reported  on  the  PartsUsage  file  with  a  value  less  than  the  minimum  are  consolidated  prior 
to  model-build.  It  is  recommended  that  future  SOE  DST  PartsUsage  files  contain  this 
data  in  order  to  create  a  more  standardize  model-build  process. 
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V.  B-OLT-DRIVEN  MODELS  ASSESSMENT 


We  need  to  assess  the  quality  of  the  models  built  using  B-OLT.  This  assessment 
involves  checking  the  model’s  ability  to  produce  reasonable  results  based  on  consistent 
inputs.  Once  assessed,  a  validation  of  the  model  may  begin.  The  following  describes  the 
assessment  and  validation  of  the  B-OLT -built  models. 

A.  VERIFICATION/ASSESSMENT:  MODELED  FAILURES  COMPARED 
TO  DATA  HISTORY 

The  automated-build  process  is  assessed  using  five  separate  HMMWV  variants: 
D1001,  D1002,  D0187,  D1 125,  and  D1 159.  The  populations  used  are  those  of  the  entire 
Marine  Corps.  By  using  the  entire  Marine  Corps,  the  computed  failure  rates  can  be 
consolidated  and  are  not  directly  influenced  by  specific  environment  factors  that  may  or 
may  not  be  present  in  one  location  or  another.  A  total  of  five  models  are  built  and  run. 
The  MOE  is  each  model’s  ability  to  replicate  the  modeled 
year’s  failures. 

ModelUER-  SOEPartCount 

The  proportion  ot  difference  was  computed  with  - 

SOEPartCount 

where  Model  UER  is  the  modeled  failures  and  SOEPartCount  are  the  actual  failures.  The 
results  are  presented  in  Figure  9.  The  chart  displays  the  distribution  of  the  difference 
between  modeling  and  actual  failures.  The  larger  bars  demonstrate  more  occasions  when 
that  particular  value  was  reported.  The  percent  difference  distribution  figure  shows  that 
the  average  percent  difference  is  around  30%.  With  an  average  of  30%  difference 
between  real  and  actual  failures,  the  results  seemed  disappointing.  There  was  reason  for 
concern  with  vehicle  populations,  and  adjustments  were  made. 
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Figure  9.  The  distribution  of  proportion  difference  between  the  actual  failures  and 

modeled  failures. 


The  outliers  tend  to  be  those  parts  with  a  high  number  of  failures  and,  because  the 
part  counts  directly  impact  the  failure  rate  in  SOE  DST,  these  are  also  the  parts  with  a 
higher  failure  rate.  The  D1 125  and  D1 159  have  the  largest  populations’  numbers  and  the 
largest  standard  deviations.  Given  that  the  vehicle  populations  come  from  the  serialized 
count  document  from  SOE  DST,  but  the  E1RR  computation  comes  from  an  adjusted 
vehicle  population  based  on  rules  within  SOE  DST,  it  is  reasonable  to  assume  the  vehicle 
populations  are  misrepresented  in  the  first  draft  of  the  model. 

The  average  percentage  difference  for  D1125  and  D1159  was  32%  and  30%, 
respectively,  while  the  percent  difference  for  DO  187  was  11%.  Because  SOE  DST  does 
not  contain  the  number  of  vehicles  used  to  calculate  failure  rate,  it  is  possible  the  original 
model  contained  an  inappropriate  number  of  vehicles.  The  D1125  and  D1159  models 
were  re-created.  The  vehicle  populations  for  D1 125  and  D1 159  were  reduced  by  20%  of 
their  original  count  and  a  better-assessed  model  resulted.  Figure  10  shows  the  new 
distribution  of  proportion  differences  that  result  from  the  adjusted  populations.  Once  a 
reasonable  percent  difference  was  achieved,  actual  difference  numbers  were  examined. 
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Figure  10.  Proportion  difference  on  adjusted  vehicle  population  actual  failures  v. 

modeled  failures. 


For  each  variant,  the  actual  difference  between  real  and  model  parts  failures  was 
computed.  The  computation  for  literal  difference  ( ModelUER  -  SOEPartCount )  results 
in  a  better  picture  of  what  the  real  difference  is  between  the  models’  failures  and  real 
failures.  Figure  1 1  shows  the  results  taken  from  all  TAMCNs  at  the  Marine  Corps 
inventory  levels  after  D1 125  and  D1 159  vehicle  counts  are  adjusted.  For  a  total  of  1,3 15 
parts,  the  models  had  a  mean  actual  difference  of  0.53.  This  mean  difference  is  the 
average  difference  of  modeled  failures  from  real  failures.  Given  that  the  tool  will  be  used 
to  project  maintenance  parts  requirements,  the  overestimation  is  desirable.  Smart  over¬ 
provisioning  will  ensure  parts  are  available  for  repairs,  while  not  stressing  embarkation  or 
budget  restraints. 


35 


Distributions 


Difference 


cr  ■  3 

l_l  - - 

r 

V . 

0 

10 

The  outliers  (circled  in  red) 
represent  high  vehicle 
populations  with  high  parts 
failures  in  SOE  DST  data. 


Quantiles 


100.0% 

maximum 

14.05 

99.5% 

6.12 

97.5% 

3.11 

90.0% 

1.24 

75.0% 

quartile 

0.58 

50.0% 

median 

0.25 

25.0% 

quartile 

0.10 

10.0% 

-0.01 

2.5% 

-0.10 

0.5% 

-0.24 

0.0% 

minimum 

-0.43 

Moments 

0.5399772 
0.9798232 
0.02702 
0.5929842 
0.4869702 
1315 


Mean 
Std  Dev 
Std  Err  Mean 
upper  95%  Mean 
lower  95%  Mean 
N 


Figure  1 1.  Distribution  of  Model  Failures-SOE  DST  failures  across  TAMCNs  in  the 
Marine  Corps  total  population.  The  model  is  overestimating  failure  events 

in  general. 


While  the  outliers  are  of  concern,  the  overall  results  are  promising.  The  outliers 
are  all  parts  with  high  part  counts  and  high  vehicle  population  counts.  However,  having  a 
model  that  assesses  this  well  would  provide  a  practical  tool  to  answer  “what  if’ 
questions.  For  example,  if  the  Marine  Corps  had  a  chance  to  switch  vendors  for  a 
particular  part  that  promised  a  better  failure  rate,  the  effects  of  that  adjustment  could 
be,  at  least,  roughly  quantified. 

Figure  12  shows  the  correlation  between  the  URR,  part  count,  and  difference  in 
the  models.  Correlation  is  the  degree  of  linear  association  between  factors  (Devore, 
2008).  Here,  a  correlation  of  0.71  suggests  that  high  URRs  are  associated  with  high 
differences  between  modeled  and  real  parts  failures.  As  more  failures  accrue,  the 
distance  between  reality  and  model  increases. 


Correlations 

URR  Part  Count  Difference 
URR  1.0000  0.7092  0.7040 

Part  Count  0.7092  1.0000  0.9958 

Difference  0.7040  0.9958  1.0000 


Figure  12.  The  correlations  between  URR,  Part  Count,  and  Difference  across 

all  TAMCNs. 


36 


B. 


FURTHER  EXPLORATION  OF  ALL  REGIONAL  ACTIVITIES  CODES 
MODEL 


Once  reasonable  models  were  developed,  regression  analysis  was  conducted.  For 
the  analysis,  the  removal  rate,  vehicle  count,  annual  miles,  and  average  miles  per  hour 
were  considered.  The  output  analyzed  is  the  difference  between  actual  and  modeled 
failures.  The  analysis  is  done  in  JMP®  using  a  step-wise  regression. 

Sorted  parameter  estimates,  like  the  ones  in  Figure  13,  present  the  factors  in 
descending  order,  from  most  to  least  significant.  The  significance  is  measured  with  the  t- 
ratio.  An  absolute  value  t-ratio  of  greater  than  two  rejects  the  hypothesis  that  the  factor  is 
zero  (Devore,  2008).  If  the  factor  is  zero,  it  is  meaningless  in  the  computation  of  the,  in 
this  case,  failures  recorded  and  subsequent  difference  from  reality. 


Sorted  Parameter  Estimates 


Term 

URR*VC 

URR*VC*Annual  Miles 
URR 

URR*Annual  Miles 
VC 

VC*Annual  Miles 
Annual  Miles 


Estimate 

Std  Error 

14.20721 

0.312745 

-0.006236 

0.000139 

-1202.839 

29.02425 

0.2316148 

0.007091 

-0.002222 

0.004231 

9.8032e-7 

1.873e-6 

-8.957e-6 

0.000108 

t  Ratio 


Profc»|t| 

<.0001* 

<.0001* 

<.0001* 

<.0001* 

0.5995 

0.6008 

0.9337 


Figure  13.  Parameter  estimates  for  all  RACs  data  for  all  TAMCNs. 


When  the  SOE  DST-driven  model  is  verified,  there  are  interaction  factors  relevant 
to  the  outcome.  Correct  URR  figures  allow  the  model  to  be  properly  affected  by  other 
factors,  such  as  the  number  of  vehicles  with  those  parts  and  the  miles  driven  by  those 
vehicles.  Two  points  may  be  made: 

•  The  wrong  URR  will  cause  the  model  to  drift  away  from  reality. 

•  If  the  correct  URR  is  captured,  the  number  of  vehicles  and  miles  driven 
impact  the  assessment  strength  of  the  model. 

As  the  number  of  parts  that  failed  during  the  period  increases,  so  does  the 
difference  between  what  failed  in  the  model,  as  compared  to  reality.  There  are  pairwise 
correlations  that  explain  why  URR  is  the  most  significant  factor.  When  SOE  DST 
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computes  failure  rates,  it  is  a  function  of  uptime  and  the  number  of  parts  that  have  failed 
during  the  given  period.  Uptime  is  detennined  by  vehicle  and  is  the  number  of  days  a 
serial  number  is  available  for  operation.  Correlations  between  VC  and  annual  miles 
driven  are  reasonable.  The  correlations  show  that  there  is  interaction  between  these 
factors  and  the  model  makes  sense.  The  positive  correlation  indicates  that  as  the  URR, 
miles  driven,  and  vehicle  counts  increase  in  reality,  the  deviation  from  reality  is  expected 
to  increase.  Since  URR  is  a  function  of  the  number  of  parts  that  have  failed  during  the 
given  period,  increased  numbers  of  failures  means  an  increase  in  deviation  from  reality 
within  the  model  as  well.  Finally,  these  all  demonstrate  the  importance  of  the  URR  (e.g., 
failure  rate)  within  SOE  DST  being  as  accurate  as  possible. 

C.  STEPS  TOWARD  VALIDATION:  PREDICTIONS  MADE  BY  VERIFIED 

MODELS 

In  2003,  I  MEF  deployed  to  Iraq  with  a  portion  of  their  legacy  JLTV  assets.  The 
MEF’s  overall  vehicle  population  increase  was  known.  VAMOSC  provides  a  deployed 
and  garrison  annual-miles  driven  rate.  SME  input  suggests  the  speed  of  deployed 
vehicles  increased  to  an  average  of  30  mph.  SOE  DST-driven  TLCM-AT  models  were 
built  using  B-OLT  and  2002  data.  These  models  were  then  run  by  increasing  the  vehicle 
counts  and  the  operational  tempo  within  TLCM-AT — with  surprising  results. 

1.  Known  Changes  Key  in  Testing  Prediction  Capability 

For  meaningful  validation,  the  results  that  the  model  is  to  be  compared  with  must 
be  known,  but  not  used  to  build  the  model.  Data  for  2003  were  not  used  in  the  creation 
of  the  2003  models;  however,  to  test  the  model’s  predictive  capacity,  it  must  be  presumed 
that  some  elements  would  be  known.  For  this  research,  the  following  was  assumed  to  be 
known  in  2002: 

•  2002  SOE  DST  data. 

•  Vehicle  numbers  for  the  deployment. 

•  An  estimate  for  miles  to  be  driven  in  2003. 

•  An  estimated  average  speed. 
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In  2002,  I  MEF  was  given  the  mission  to  deploy  to  Iraq.  Prior  to  leaving,  they 
received  an  increase  in  equipment  allocations  and  could  estimate  that  their  operational 
tempo  would  increase.  For  this  research,  VAMOSC  was  used  to  get  average  annual 
miles  for  deployed  legacy  JLTV  assets.  The  MDR  was  used  to  get  reporting  quantities  of 
legacy  JLTVs  for  the  MEF  during  2003.  Both  data  elements  would  have  been  known,  or 
could  have  been  estimated,  in  2002. 

The  2002  model  was  built  and  assessed.  The  known  changes  (count,  miles,  mph) 
were  then  changed  in  the  assessed  model.  In  so  doing,  the  SOE  DST-driven  TLCM-AT 
model,  built  using  B-OLT,  was  able  to  “predict”  2003. 

2.  The  Model  Demonstrated  an  Increase  in  Failures  Based  on  Known 
Expected  Operational  Tempo  Changes 

The  B-OLT  model  was  subject  to  some  limitations.  As  discussed  earlier,  SOE 
DST  only  reports  on  failures  that  occurred  during  the  time  frame  requested.  Therefore,  in 
the  2002  to  2003  prediction,  there  were  parts  that  failed  in  2002  that  did  not  fail  in  2003. 
Additionally,  there  are  multiple  NSNs  that  represent  the  same  part.  These  two  facts 
combined  made  it  impossible  to  thoroughly  investigate  the  model’s  predictive  capacity. 
Prior  to  models  being  built,  2002  and  2003  PartsUsage  files  were  compared  and  only 
those  NSNs  in  common  between  the  two  years  were  modeled. 

Given  the  limitations,  there  were  still  insights  to  be  gained  by  comparing  the 
predicted  failures  with  the  failures  experienced  during  2003.  The  NSNs  that  failed  in 

2002  were  modeled  in  the  2003  model.  All  modeled  failures’  NSNs  were  compared  to 

2003  SOE  DST  PartsUsage  file  Parts  Count.  There  were  297  total  parts  that  were 
modeled  across  the  TAMCNs.  From  these  297  parts,  the  mean  absolute  difference  was 
2.52.  Again,  this  difference  was  a  conservative  estimate,  and  would  ensure  that  more 
parts  are  available  for  repairs.  The  correlations,  in  Figure  14,  demonstrate  again  the 
reliance  on  the  URR  and  total  failures. 
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Correlations 


UER  Total 
UER  Total  1 .0000 

URR  0.7949 

Difference  0.7329 


URR  Difference 

0.7949  0.7329 

1.0000  0.7994 

0.7994  1.0000 


Figure  14.  The  correlation  coefficients  of  the  number  of  part  failures,  URR,  and  the 
difference  between  model  and  reality.  The  prediction  exercise  shows  promise  and  the 
correlations  suggest  where  improvement  can  be  made. 

Given  the  current  predictions  from  the  model,  as  built  from  existing  SOE  DST 
data,  a  provisioning  package  could  be  made.  Specifically,  the  UER  (modeled  failures) 
quantities  could  be  used  to  draw  a  Class  IX  block.  If  the  UER  quantities  were  taken  plus 
one,  two,  or  three  standard  deviations,  the  results  would  be  as  pictured  in  Figure  15. 
Along  the  x  axis  are  the  results  of  UER-Parts  Count  computation.  The  results  show  that 
going  just  one  standard  deviation  more  than  the  model’s  prediction  would  have  resulted 
in  being  short  on  only  17  of  297  parts  during  the  provisioning  period. 


Results  of  Provisioning  with 
Predictive  Model 


■  1SD 

■  2SD 

■  3SD 


Figure  15.  The  results  of  provisioning  with  predictive  model.  The  columns  represent 
one,  two,  or  three  standard  deviations  above  the  modeled  requirement  minus  the  actual 

requirements  in  2003. 

Until  SOE  DST  reports  failure  rates  for  all  components  of  the  PEI,  and  multiple 
NSNs  can  be  linked  to  a  common  part  number,  true  measures  of  predictability  will  be 
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impossible.  The  limited  tests,  while  promising,  are  just  a  first  step  toward  validating  and 
using  the  B-OLT  models  as  a  predictive  tool.  To  prepare  for  the  eventuality  of 
employing  these  tools  in  a  predictive  capacity,  it  was  reasonable  to  test  the  sensitivity  of 
the  B-OLT  models  to  the  factors  that  were  not  directly  obtained  from  SOE  DST. 

With  assurance  that  the  model  was  acting  as  expected,  a  logical  next  step  is  to 
explore  sensitivities  to  those  factors  that  are  being  estimated  in  the  model.  For  the 
SOE  DST-driven  TLCM-AT  model,  those  factors  are: 

•  Annual  miles  driven. 

•  Vehicle  count  for  each  population. 

•  Average  MPH. 

•  Probability  that  a  part  will  not  be  repairable. 

•  Unexpected  remove  rate. 

The  next  step  is  to  develop  a  DOE  to  saturate  the  design  space  in  a  most  efficient 
manner  to  test  sensitivities  to  the  factors  of  interest  listed  above. 
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VI.  MODELS’  SENSITIVITY  TO  INITIAL  FACTORS 


During  the  assessment  of  the  models,  factors  were  identified  that  might  influence 
the  MOE.  These  factors  are  explored  using  DOE.  The  DOE  methodology  is  explained 
and  results  presented. 

A.  NEARLY  ORTHOGONAL  LATIN  HYPERCUBE  (NOLH)  DESIGN 
EXPLANATION 

The  NOLH  DOE  is  capable  of  saturating  a  design  space  to  discover  factors  of 
importance.  To  test  a  response  variable’s  sensitivity  to  specific  factors,  a  design  should 
be  made  to  ensure  little  or  no  correlation  between  factors.  In  making  the  design  points 
nearly  orthogonal  through  an  NOLH  DOE,  the  estimates  of  the  coefficients  in  the 
associated  regression  models  are  uncorrelated  (Cioppa  &  Lucas,  2007).  The  model  sets 
described  in  this  chapter  were  all  created  using  a  NOLH  design. 

B.  GENERAL  DOE  FOR  SENSITIVITY  ANALYSIS  FOR  FACTORS  USED 
IN  THE  MODEL-BUILD 

An  efficient  DOE  was  developed  using  NOLH  designs.  These  DOEs  were  run 
using  the  SOE  DST  PartsUsage  file  pulled  for  TAMCNs  D0187,  D1001,  and  D1002. 
Records  for  all  Regional  Activity  Codes  (RACs)  in  the  Marine  Corps  were  used. 
Accordingly,  there  were  three  model  sets  developed  in  the  research  associated  with  this 
portion  of  the  thesis. 

There  were  some  common  design  factor  ranges  for  all  legacy  JLTV  variants  and 
they  are  displayed  in  Table  3  as  they  were  used  in  the  NOLH  design. 

Table  3.  NOLH  DOE  factors  that  are  common  between  all  TAMCNs. 


ProbCon 

MPH 

RateChange 

Low 

0.1 

20.0 

0.5 

High 

0.99 

50.0 

2.0 

Probability  of  condemnation  (ProbCon)  is  the  probability  that  a  repairable  part 
will  not  be  repairable  and  will  need  to  be  replaced.  As  discussed  earlier,  the  URR  is 
derived  from  the  SOE  DST’s  failure  rate  and  is  equivalent  to  the  inverse  of  Mean  Time 
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Between  Failure  (MTBF).  In  order  to  vary  this  within  the  DOE,  each  URR  was 
multiplied  by  a  unifonn  rate  change  between  0.5  and  2.0.  Finally,  mph  was  varied  from 
20  to  50,  based  on  SME  input. 

The  annual  miles  driven  and  the  vehicle  population  factors  were  also  adjusted  in 
the  DOE.  The  high  and  low  values  for  these  ranges  were  dependent  on  the  original  value. 
For  annual  miles,  VAMOSC  provided  the  average  miles  driven  in  garrison  environments. 
This  figure,  by  TAMCN,  was  adjusted  to  plus  or  minus  1,000  and  is  used  as  the  high  and 
low  of  the  range,  respectively.  Vehicle  populations  were  taken  from  the  SOE  DST’s 
serialized  location  report.  The  figures  provided  by  SOE  DST  where  then  adjusted  plus  or 
minus  30%  for  the  high  and  low  of  the  range,  respectively. 

TLCM-AT,  when  defining  the  URR,  allows  the  user  to  incorporate  two 
parameters  for  the  Weibull  distribution.  If  a  shape  other  than  one  is  used  and  the  MTBF 
is  also  adjusted,  the  result  is  an  unclear  representation  of  the  effects  of  the  shape  on  the 
MOE.  In  general,  if  an  exponential  distribution  is  used  instead  (Weibull  with  shape  =  1), 
the  result  will  be  a  distribution  with  wider  variation.  Since  the  worst-case  scenario  is 
acceptable,  the  shape  parameter  was  not  used  as  a  factor  of  possible  interest. 

C.  SENSITIVITY  RESULTS 

For  DO  187,  17  design  points  were  run  with  vehicle  counts  that  represented  the 
population  for  all  RACs.  Each  design  was  run  for  a  total  of  100  histories.  A 
5-factor  design  can  be  executed  with  just  17  design  points.  The  NOLH  was  set  up,  and 
the  model  was  run,  using  the  initial  17  design  points.  An  example  of  the  design  can  be 
seen  in  Figure  16.  The  data  in  green  are  the  user’s  inputs.  Each  row  below  corresponds 
to  a  given  design  point.  The  columns  contain  the  factor  settings  across  the  design  points. 
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low  level 

794 

187 
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20 
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VC 
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RC 
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919 
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12 

2544 
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0.38 
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0.33 

46 

1.5 

15 

794 

297 

0.35 

26 

1.9 

16 

1544 

317 

0.1 

41 

0.7 

17 

1669 

207 

0.2 

22 

0.8 

Figure  16.  The  NOLH  DOE  produced  for  the  DO  187. 


The  IDFW  1 8  group  decided  that  the  best  MOE  when  assessing  the  model-build 
process  was  the  model’s  deviance  from  reality  in  terms  of  parts  failures.  This  translated 
into  a  RMSD  computation. 

For  each  design  point,  there  is  a  root  mean  square  error  computed  for  the 
difference  between  how  many  parts  actually  failed  and  how  many  parts  failed  in  the 


model.  Root  mean  squared  difference  (RMSD)  = 


y,  ( ModelUER  -  SOEPartCouniy 


1 


parts 


where  Model  UER  are  the  modeled  failures,  and  SOE  DST  PartCount  are  actual  failures 
for  the  given  time  period,  model,  and  population.  This  difference  is  squared  and 
normalized,  and  the  sum  taken  from  across  all  parts  associated  to  that  TAMCN.  This 
figure  allows  an  overall  MOE  of  that  design  point  to  return  an  expected  result  according 
to  how  it  compares  to  the  actual  failures  experienced  in  reality. 
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Initial  results  were  not  promising.  The  model-build  process  rolls  up  all  parts  that 
cost  less  than  $10  into  a  single  part  line  number.  This  is  done  to  ensure  that  computation 
limits  are  not  exceeded.  Remember  that  TLCM-AT  populates  an  event  list  based  on  the 
failure  rates  of  each  LRU.  With  no  indenture  structure  provided  in  SOE  DST,  the  models 
built  for  this  thesis  are  LRU-only  models.  During  the  post  analysis,  it  was  discovered 
that  the  line  item  for  the  consolidated  parts  contributed  a  lot  of  variability  into  the  model. 
This  is  a  result  of  the  wide  range  of  failure  rates  associated  to  the  individual  parts  that 
make  up  this  consolidated  parts  line.  Once  this  line  was  deleted  from  the  analysis,  the 
RMSD  analysis  made  more  sense. 

Correlations  again  were  examined  first  to  detennine  which  factors  were  correlated 
with  the  RMSD.  In  Figure  17,  the  following  abbreviations  apply: 

•  AM:  Annual  miles,  the  miles  driven  during  the  simulation. 

•  VC:  Vehicle  Count,  the  number  of  vehicles  in  the  simulation. 

•  ProbCon:  The  probability  of  a  repairable  part  being  condemned. 

•  MPH:  Miles  per  hour. 

•  RC:  Ratio  Change.  The  computed  URR  is  multiplied  by  this  factor  to 
represent  an  incorrect  URR. 

•  RMSD:  Root  mean  squared  difference. 
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0.6487 

1.0000 

Figure  17.  The  correlations  in  the  NOLH  DOE  performed  on  the  DO  187  variant.  The 
positive  correlation  between  RMSD  and  RC  indicate  that  an  increase  in  RC  will  result  in 
a  larger  overall  change  in  the  model’s  ability  to  back  validate  well. 


The  correlations  are  displayed  to  demostrate  the  orthogonal  nature  of  all  the  input  factors. 
This  is  a  result  of  the  careful  DOEs  using  an  NOLH.  It  can  be  seen  that  the  individual 
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factors  do  not  share  correlation.  Corrrelation  between  a  factor  and  RMSD  demonstrates 
that  factor’s  linear  effect  on  the  eventual  outcome. 

D1001  and  D1002  were  run  through  the  same  fundamental  DOE.  The  only 
change  was  that  a  stacked  design  was  used,  so  there  were  34  total  design  points  for  the 
D1001  and  D1002  models.  The  correlation  matrices  are  in  Figure  18.  The  URR  remains 
a  factor  in  all  models;  however,  the  degree  of  influence  varies  considerably  between 
the  models. 
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Figure  18.  The  correlation  coefficients  for  the  D1001  (left)  and 
D1002  (right)  variants. 


The  results  were  then  all  consolidated  and  analyzed  together.  The  correlations  in 
Figure  19  demonstrate  the  overall  correlations  between  all  TAMCNs  modeled.  The 
TAMCN  is  a  categorical  variable  applied  to  the  different  variants.  Vehicle  counts  and 
TAMCN  are  negatively  correlated  because  there  are  more  vehicles  in  the  D1001  than  are 
in  the  DO  187,  by  a  factor  of  five. 


Correlations 


TAMCN 

AM 

VC 

ProbCon 

MPH 

RC 

RMSD 

TAMCN 

1.0000 

-0.2924 

-0.7144 

-0.0005 

-0.0002 

-0.0001 

0.1344 

AM 

-0.2924 

1 .0000 

0.1965 

0.0001 

0.0017 

0.0244 

-0.0332 

VC 

-0.7144 

0.1965 

1.0000 

0.0006 

0.0029 

0.0027 

-0.4180 

ProbCon 

-0.0005 

0.0001 

0.0006 

1.0000 

-0.0087 

0.0183 

-0.0191 

MPH 

-0.0002 

0.0017 

0.0029 

-0.0087 

1 .0000 

-0.0206 

0.0022 

RC 

-0.0001 

0.0244 

0.0027 

0.0183 

-0.0206 

1.0000 

0.0388 

RMSD 

0.1344 

-0.0332 

-0.4180 

-0.0191 

0.0022 

0.0388 

1 .0000 

Figure  19.  Correlation  coefficients  for  all  variants  together. 

D1001,  as  can  be  seen  in  Table  4,  has  the  smallest  number  of  vehicles,  yet  the 
largest  number  of  parts  per  vehicle  in  the  model.  With  D1001,  the  RC  factor  and  the 
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vehicle  count  have  a  larger  impact  on  the  overall  RMSD.  The  importance  of  accessibility 
to  accurate  and  thorough  input  data  with  modeling  becomes  more  apparent  given  the 
impact  that  population  and  number  of  parts  per  platform  have  on  the  assessment  of  the 
model.  These  factors  change  with  every  MEF  and  for  every  TAMCN;  thus,  reusing  an 
existing  model  is  not  as  appealing  as  employing  an  automated  model-building  process. 

Table  4.  Design  of  experiments  with  the  addition  of  the  Parts  Count  column.  The  Parts 

Count  is  the  number  of  parts  modeled  per  vehicle. 


Factors: 

AM 

VC 

ProbCon 

MPH 

RC 

Parts  Count 

low  level 

794 

187 

0.1 

20 

0.5 

D0187  high  level 

2794 

347 

0.5 

50 

2 

69 

low  level 

802 

49 

0.1 

20 

0.5 

D1001  high  level 

1802 

93 

0.5 

50 

2 

174 

low  level 

279 

106 

0.1 

20 

0.5 

D1002  high  level 

2279 

198 

0.5 

50 

2 

146 

The  apparent  impact  of  parts  to  vehicle  population  ratio  demonstrates  a 
requirement  to  rethink  how  SOE  DST  displays  output.  Each  legacy  JLTV  variant  should 
contain  more  or  less  the  same  amount  of  parts.  If  SOE  DST  produced  a  by-part-number 
breakdown  structure  for  a  TAMCN,  concern  about  the  effects  of  parts  density  per  vehicle 
in  the  model  would  be  avoided.  Additionally,  a  better  understanding  of  all  parts  would 
be  achieved. 

D.  FACTORS  OF  INTEREST  BASED  ON  SENSITIVITY  ANALYSIS 

For  initial  sensitivity  analysis,  the  scope  was  limited  to  DO  187.  Here  the  factors 
were  regressed  against  the  RMSD.  Because  of  the  NOLH  DOE,  true  significant 
interactions  are  discovered. 

For  DO  187  populations  in  all  RACs,  the  DOE  factors  used  accounted  for  95%  of 
the  variability.  The  parameter  estimates  are  shown  in  Figure  20.  DO  187  had  the  largest 
vehicle  population  and  the  lowest  parts  per  vehicle  modeled. 
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Sorted  Parameter  Estimates 


Term 

VC*RC 

VC 

RC 

AIVTProbCon 

AM 

ProbCon 

ProbCon*RC 


Estimate 

0.0411483 

-0.040825 

-9.610538 

-0.005625 

0.0016249 

6.7758102 

2.9365682 


Std  Error 

0.005684 

0.007457 

1.771039 

0.001816 

0.000578 

3.383709 

2.157173 


t  Ratio 

7.24 

-5.47 

-5.43 

-3.10 

2.81 

2.00 

1.36 


Prob>|t| 

<.0001* 

0.0004* 

0.0004* 

0.0128* 

0.0204* 

0.0762 

0.2065 


Figure  20.  The  parameter  estimates  for  DO  187  on  16  design  points.  The  R-Squared 
for  the  model  is  95%,  with  an  adjusted  of  92%.  The  model  fits  well. 


D1001  models  well,  with  an  R-Squared  of  0.75  and  adjusted  of  0.73.  This 
indicates  that  there  is  a  reasonable  fit  with  the  factors  involved;  however,  the  actual 
factors  of  interest  are  much  different  than  those  of  DO  187.  For  D1001,  the  interaction 
between  RC  and  VC  tends  to  be  more  significant  than  does  RC  alone,  as  seen  in 
Figure  21.  DlOOl’s  PartsUsage  file  has  a  small  ratio  of  parts  to  vehicles.  Specifically, 
for  the  low  and  high  vehicle  counts  in  the  NOLH  design,  there  are  between  3.5  and  1.87 
vehicles  per  part  modeled. 


Sorted  Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

VC*RC 

-0.018921 

0.000809 

-23.38 

1  ! 

<.0001* 

RC 

0.1496515 

0.058465 

2.56 

] 

0.0105* 

VC 

-0.000175 

0.001074 

-0.16 

0.8706 

Figure  21.  The  parameter  estimates  for  DO  187  on  32  design  points  across  174  parts. 
The  R-Squared  for  the  model  is  75%  with  and  adjusted  of  75%. 


D1002  has  a  large  vehicle  population  and  a  large  number  of  parts  per  vehicle. 
The  two  combined  to  produce  a  lot  of  variability  in  the  model.  For  D1002,  there  are 
between  1.38  and  0.73  vehicles  per  part  in  the  models.  The  extreme  points  are  associated 
with  design  points  on  the  large  end  of  the  population  and  ratio  change.  In  that  situation, 
there  are  many  vehicles,  each  with  146  parts  and  those  parts  are  failing  at  a  faster  rate. 
This  naturally  will  cause  a  large  deviation  from  the  real  number  of  parts  failures 
experienced  by  this  population.  The  R-squared  for  the  D1002  model  is  0.22,  with  an 
adjusted  0.17.  With  this  R-Squared,  the  model  is  not  suited  for  use  in  “what  if’  analysis. 
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Given  the  extreme  differences  in  the  three  models  exposed  to  the  DOE,  there  is 
little  collected  insight  to  be  gained  for  the  maintenance  process  alone.  However,  for  all 
models,  the  ratio  change  and  vehicle  counts  were  main  contributors  to  variability.  While 
the  R-squared  values  are  considerably  different  between  the  models,  there  are  great 
differences  in  the  structure  of  the  models.  The  difference  in  the  models  is  a  result  of  SOE 
DST  data  structure.  A  universal  approach  to  the  SOE  DST  data  would  provide  continuity 
in  TLCM  models  in  the  future.  Specifically,  the  following  changes  would  help: 

•  Indenture  structure  for  the  TAMCN  would  ensure  all  its  parts 
were  reported. 

•  Work  Unit  Code  or  Logistics  Control  Number  and  Part  Number  in 
addition  to  NSN  reporting  will  ensure  parts  failures  are  captured,  rather 
than  simply  NSN  failures. 

•  A  complete  list  of  parts  associated  with  the  TAMCN  and  its  current  failure 
rate  will  allow  M&S  to  reflect  all  parts,  not  just  those  that  failed  during  the 
reporting  period  drawn. 

•  The  end  item  population  used  to  compute  failure  rates  in  the  SOE  DST. 

With  these  changes,  a  common  picture  of  the  TAMCNs  could  be  achieved.  Once 
done,  DOE  could  lead  to  insights  about  factors  affecting  any  number  of  measures  of 
effectiveness.  The  purpose  of  this  research  was  to  develop  the  capability  to  conduct  DOE 
analysis.  Unfortunately,  given  the  considerable  differences  between  TAMCNs,  possibly 
based  on  simple  vehicle-to-parts  ratios,  the  results  are  not  able  to  be  fully  analyzed. 
While  things  may  be  said  about  individual  TAMCNs,  it  is  expected  that  a  true  picture  of 
RMSD  will  not  be  possible  without  a  complete  (all  parts  modeled)  model. 


50 


VII.  INSIGHTS  GAINED  USEFUL  TO  FUTURE  TLCM  EFFORTS 


Many  insights  pertaining  to  the  Marine  Corps  TLCM  process  were  gained  from 
this  research.  Those  insights  have  led  to  a  realization  that  TLCM  models  can  be 
developed  with  the  data  that  are  currently  being  gathered  in  an  automated  manner.  This 
automation  of  the  process  can  lead  to  validation  of  the  model-build  process  and  put 
modeling  in  the  hands  of  PMs  to  assist  in  policy  decisions.  Finally,  conceptual  future 
directions  and  opportunities  to  continue  research  were  discovered. 

A.  VERIFICATION  OF  THE  AUTOMATED-BUILD  PROCESS 

This  research  originally  was  designed  to  explore  a  model  created  by 
Clockwork  Solutions’  contractors  within  the  TLCM- AT  environment.  While  developing 
the  background  for  the  work,  it  was  determined  there  is  practicality  in  streamlining  the 
model-build  process  in  order  to  make  maintenance  modeling  more  accessible.  This 
research  built  an  automation  tool  that  was  proven  capable  of  taking  collected  data  and 
building  adequate  models.  Prior  to  B-OLT,  a  rudimentary  model  built  strictly  from  SOE 
DST  and  limited  SME  data  took  approximately  three  days  by  a  trained  TLCM-AT  user. 
The  models  can  now  be  built,  run,  and  output  extracted  in  less  than 
10  minutes. 

There  are  more  steps  that  must  be  taken  to  build  thorough  models.  Future  work 
must  be  applied  to  discovering  what  data  we  should  be  collecting  and  how  to  automate 
data  collection.  Specifically,  for  TLCM-AT  models,  there  are  modules  that  are  not 
populated  with  B-OLT.  By  incorporating  more  data  in  SOE  DST  and  expanding  B-OLT 
a  more  thorough  TLCM-AT  model  may  be  built.  A  more  complete  model  will  allow  for 
greater  opportunities  with  modeling  and  simulation  (M&S)  in  TLCM. 

B.  ASSESSMENT  OF  B-OLT-GENERATED  MODELS 

B-OLT  drew  data  from  SOE  DST  and  verified  that  the  models  built  in  this 
automated  fashion  acted  as  they  should.  There  was  some  variance  from  reality  when 
back-assessing  the  models.  These  variances  led  to  examination  of  the  factors  of  interest 
of  those  variables  captured  in  the  model. 
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C.  INITIAL  FACTORS  OF  INTEREST  IDENTIFIED  USING  SOE  DST  AND 

TLCM-AT 

Future  work  can  lead  to  a  better  understanding  of  factors  of  interest  in  TLCM 
MOEs.  B-OLT  has  made  it  possible  to  automatically  build  and  extract  data  from 
TLCM-AT  using  data  from  SOE  DST.  SOE  DST  does  not  provide  a  complete  vehicle 
structure  to  model  in  TLCM-AT.  While  TLCM-AT  can  be  used  with  as  many  or  as  few 
parts  as  the  user  would  like,  quantitative  analysis  on  TLCM  effects  requires  some 
continuity  between  TAMCNs  modeled.  However,  it  is  clear  even  from  this  research  that 
the  failure  rate  and  vehicle  counts  are  major  contributors  to  RMSD. 

D.  SIGNIFICANCE  OF  AUTOMATED-BUILD  PROCESS 

B-OLT  provides  TLCM  professionals  with  an  opportunity  to  systematically  build 
models  using  the  data  synthesized  by  SOE  DST.  The  automation  allows  the  model-build 
process  to  be  repeated — a  critical  step  in  model  vcrification/validation  efforts. 
Additionally,  the  B-OLT  may  be  applied  across  TAMCNs. 

1.  Time  Involved  in  Manually  Building  TLCM-AT  Models  is 
Considerable 

Building  a  TLCM-AT  model  was  at  least  a  three-day  process  for  a  trained  user 
prior  to  B-OLT.  It  is  acknowledged  that  the  models  built  today  with  B-OLT  are  not  as 
robust  as  a  three-day  model-build  using  SMEs  and  multiple  data  sources;  however,  it  is 
not  far  off.  The  reality  is  that  this  automated  model-build  technique  may  be  refined  and 
improved,  and  then  applied  across  TAMCNs.  It  will  take  further  exploration  of  SME 
factors,  incorporation  of  TLCM  requirements  into  the  SOE  DST,  and  analysis  of  models 
to  determine  what  factors  are  important.  Once  detennined,  the  next  step  must  be 
exploring  how  we  can  automate  the  data-gathering  step.  No  model  will  be  better  than  the 
data  it  is  built  on  and  a  commitment  to  TLCM  demands  a  commitment  to  data  collection. 

2.  Opportunity  for  Use  by  All  PMs  for  All  TAMCNs 

PMs  that  are  required  to  develop  policy  to  improve  mission  availability,  make 
projections  of  maintenance  requirements,  and  ascertain  the  overall  benefit  of  product 
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shifts  (either  PEI  or  component  level)  must  be  able  to  simulate  the  maintenance  cycle. 
To  do  this,  the  model  purchased  by  the  Marine  Corps  must  be  accessible  by  the  PM.  It  is 
acknowledged  that  simulations  are  not  for  the  uninformed;  however,  a  user-friendly 
model-build,  coupled  with  TLCM-AT’s  ability  to  accept  one  or  two  variable  changes, 
lends  itself  to  quick,  effective  “what  if’  analysis.  With  that  understanding,  a  uniform 
way  of  producing  models  with  the  data  synthesized  by  SOE  DST  will  provide  model 
access  to  PMs. 


3.  Opportunity  to  Apply  DOE  to  Policy  Decisions 

The  ability  to  quickly  transfer  data  gathered  into  models  with  some  level  of 
assurance  that  they  reflect  reality  leads  to  an  opportunity  to  exercise  that  model  to  gain 
insight  on  potential  policy  decisions.  Using  a  DOE,  factors  that  affect,  or  do  not  affect,  a 
MOE  can  be  explored.  B-OLT  allows  for  quick  and  easy  model-build  and  verification. 
Additionally,  a  loop  was  put  around  the  code  to  accommodate  NOLH 
DOE  runs. 

4.  Opportunity  to  Work  toward  Model  Validation 

Validation  of  a  model  requires  control  over  the  data  used  to  build  the  model.  By 
relying  on  SOE  DST  data,  the  models  built  using  B-OLT  can  be  reused  and  there  is 
control  over  the  data  used  to  build  the  TLCM  model.  Validation  can  be  accumulated 
through  using  the  same  process  over  several  TAMCNs,  time  periods,  and  populations. 

E.  FUTURE  WORK  OPPORTUNITIES 

The  work  started  with  this  thesis  can  be  greatly  expanded  upon. 

1.  Exploration  of  Predictive  Factors  for  TLCM-AT  Airframe  Models 

TLCM- AT  was  initially  developed,  and  has  been  used,  as  a  predictive  model  with 
airframes.  Mechanical  failures  and  the  factors  that  affect  these  failures  may  be  similar 
between  air  and  ground  equipment.  An  exploratory  analysis  of  the  predictive  TLCM-AT 
airframe  models  built  using  a  DOE  may  shed  light  onto  factors  that  are  important  when 
predicting  ground  failures.  Once  this  is  determined,  these  factors  of  influence  can  be 
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gathered  and  modeled  for  ground  equipment.  If  these  factors  turn  out  to  be  significant  for 
ground  equipment,  further  work  will  be  in  order. 


2.  Cost  Benefit  Analysis  for  Automating  Data  Collection 

Throughout  the  thesis  process,  gaps  in  data  and  hazy  data  were  discovered.  Given 
the  importance  of  M&S,  a  cost  benefit  analysis  of  data  collection  automation  is 
necessary.  Specifically,  benefits  to  effective  models  must  be  quantified.  A  cursory  look 
will  demonstrate  that  deployed  units  carry  with  them  large  quantities  of  spare  parts  that 
take  up  valuable  embark  space.  If  these  parts  are  not  used  during  the  deployment,  the 
cost  of  not  having  the  capability  to  predict  failures  is  the  loss  of  embarkation  space.  This 
embarkation  space,  depending  on  the  size,  could  be  used  for  multiple  PEIs,  which  could 
be  useful  in  operational  missions.  Further,  no  matter  which  parts  are  carried  on 
deployment,  it  seems  there  are  always  parts  being  shipped  to  the  unit.  Naturally,  the  cost 
of  shipping  translates  to  a  cost  associated  to  the  unit’s  inability  to  anticipate  requirements. 
Additionally,  this  causes  down  time  for  the  PEI  and  an  overall  impact  on  the  mission.  If 
significant  savings  can  be  achieved  through  effective  modeling  and  simulation,  then  data 
collection  efforts  must  be  improved. 

3.  Further  Development  of  B-OLT  Using  Other  Data  Sources  and 
Model  Platforms 

Currently,  B-OLT  focuses  on  12  of  the  32  tables  used  in  TLCM-AT.  Further, 
B-OLT  uses  only  one  data  source — SOE  DST.  B-OLT  should  be  expanded  to  ensure  all 
of  TLCM-AT ’s  functionality  is  taken  advantage  of  in  the  future.  Identification  of 
elements  missing  in  SOE  DST  is  important,  so  they  may  be  incorporated  in  future 
versions  of  SOE  DST.  It  is  reasonable  to  estimate  that  factors  of  importance  identified  in 
TLCM-AT  models  will  be  important  in  other  TLCM  models  as  well.  As  such,  this 
identification  and  incorporation  into  SOE  DST  will  serve  the  overall  TLCM  effort. 

4.  Application  of  the  Model-Building  Process  to  Promote  its  Validation 
of  Maintenance  Models 

B-OLT -built  models  must  be  constructed  and  scrutinized  systematically  to  obtain 

a  validated  model  within  the  TLCM  community.  This  research  limited  its  scope  to  five 
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legacy  JLTV  variants.  Naturally,  other  Marine  Corps  assets  must  be  evaluated  using  the 
B-OLT-build  process.  By  gathering  these  histories,  the  TLCM  community  will  gain 
confidence  in  the  model-build  process  and  application  to  insights  aimed  at  assisting  the 
decision-making  process. 
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